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DEVELOPMENT  OF  PERFORMANCE  INDICATORS  FOR 
THE  SELECTION  AND  PROMOTION  OF  CORPS 
MANAGERIAL  TALENT 


1  INTRODUCTION 


Background 

The  Army  Corps  of  Engineers  has  traditionally  been  concerned  with  developing  effective  person¬ 
nel  selection  and  promotion  procedures,  but  only  recently  has  substantial  work  been  done  to  develop 
and  evaluate  procedures  for  identifying  and  training  managers  early  in  their  Corps  careers.  With 
increased  demand  for  technological  innovation  and  development,  the  Corps  must  persevere  in  this 
emphasis  on  the  selection  and  promotion  of  its  leadership. 

In  the  past,  problems  have  arisen  occasionally  when  a  person  --  who  apparently  had  the  proper 
background,  skills,  or  training  -  was  chosen  for  a  managerial  position,  but  failed  to  function  effectively 
in  the  job.  Typically  this  happens  when  a  person  is  promoted  because  of  experience  and  technical  com¬ 
petence,  but  lacks  the  ability  to  perform  managerial  duties.  Since  choosing  the  wrong  candidate  can 
adversely  affect  not  only  that  individual,  but  also  the  entire  organization,  better  selection  and  promotion 
decisions  may  improve  organizational  performance  by  permitting  employees  to  work  with  more 
effectiveness  and  with  less  stress  and  dissatisfaction. 

Thus,  the  research  reported  here  evolved  for  two  reasons.  The  first  was  the  Corps'  continuing 
concern  with  maintaining  the  quality  of  its  managerial  talent.  The  second  factor  is  actually  an  extension 
of  the  first  because  the  Corps  has  been  developing  and  using  selection  and  promotion  techniques  which 
have  had  varying  degrees  of  success.  Although  these  techniques  have  improved  on  previous  efforts, 
they  still  have  major  drawbacks:  most  involve  assessing  the  performance  of  individual  managers; 
assessments  tend  to  be  subjective;  all  ratings  tend  to  be  high;  and  the  techniques  reflect  too  many  fac¬ 
tors  over  which  individuals  have  no  control. 


Objective 

The  objective  of  this  study  was  to  develop  potential  performance  indicators  that  could  be  used  in 
identifying  and  selecting  top-level  managers  in  the  Corps  of  Engineers.  The  overall  objective  is  to 
improve  managerial  effectiveness. 


Approach 

Before  potential  performance  indicators  could  be  developed,  several  interrelated  issues  were  con¬ 
sidered  (11  Was  there  a  pool  of  highly  qualified  applicants  from  an  effective  career  development  pro¬ 
gram?  (2)  What  kinds  of  activities  were  important  for  success  in  higher  level  jobs?  (3)  Could  the 
appraisal  system  for  observing  and  evaluating  personnel  performance  be  improved?  (4)  Are  new  pro¬ 
cedures  needed  to  stimulate  higher  levels  of  organizational  performance? 

To  answer  these  questions,  interviews  were  conducted  with  Corps  personnel  in  one  Division 
office,  six  District  offices,  and  the  Office  of  the  Chief  of  Engineers  (OCE).  Interviewers  talked  with  a 
wide  variety  of  managers,  developed  and  reviewed  pilot  forms,  and  collected  pilot  data.  A  total  of  76 
interviews  were  held,  varying  from  30  minutes  to  2  hours  in  length  (average  about  I  hour).  The  inter¬ 
views  tended  to  focus  on  GS-14  and  -15  personnel,  but  a  number  of  GS-12s  and  -13s  were  also  inter¬ 
viewed  to  obtain  lower-level  views.  Interview  results  are  reported  in  Chapter  2. 
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Information  from  the  interviews  was  used  to  develop  eight  predictive  selection  and  promotion 
instruments  providing  data  in  four  areas.  The  first  area  focuses  on  the  job  itself  and  the  activities  of 
the  position.  Chapter  3  describes  the  Job  Activities  Description,  a  questionnaire  providing  information 
on  the  importance  of  and  time  spent  on  relevant  job  activities. 

The  second  area  is  relevant  to  the  candidates  for  an  open  position  or  promotion  and  obtains 
specific  information  about  performance  characteristics.  Three  separate  procedures  were  developed  to 
gather  such  information,  peer  rankings,  ratings  by  immediate  supervisor,  and  ratings  by  second-level 
supervisors  (Chapter  4). 

The  third  area  is  also  relevant  to  the  candidates  but  involves  information  about  potential  personal 
characteristics.  Again,  three  separate  instruments  were  designed  The  Biographical  Inventory:  Form 
V-2,  and  Biographical  Sketch;  and  the  Interviewing  Guide  (Chapters  5  and  6).  The  fourth  area  con¬ 
cerns  the  total  organization  and  the  job’s  place  within  it.  This  information  is  provided  by  the  Manage¬ 
ment  Audit  Survey,  which  assesses  the  organization’s  climate  by  surveying  all  employees  (Chapter  7). 


2  RESULTS  OF  INTERVIEWS 


This  chapter  summarizes  the  interviewees'  general  concerns  about  selection  and  promotion,  and 
their  comments  on  the  skills,  knowledge,  ability,  and  personal  characteristics  (SKAP)  system,  on 
specific  selection  strategies,  and  on  the  characteristics  of  a  good  manager. 


The  Existing  SKAP  System 

The  SKAP,  a  referral  assessment  system,  is  an  Army-wide  mechanism  for  identifying  potential 
candidates  for  positions  at  GS-14  and  above.  To  be  considered  for  promotion  or  lateral  transfer  to  such 
positions,  any  employees  at  GS-13  and  above  must  have  data  in  the  SKAP  system.  Part  of  the  data 
comes  from  assessments  of  the  individual’s  past  performance  and  potential  generated  by  the  first-level 
supervisor  and  reviewed  by  a  higher  level  supervisor  in  the  organization.  Another  portion  of  the  SKAP 
data  comes  from  the  individual.  The  employee  must  fill  out  a  form  indicating  his/her  career  goals,  the 
types  of  positions  he/she  would  be  interested  in  having,  and  geographical  locations  where  he/she  would 
be  willing  to  accept  such  positions.  The  individual  also  supplies  personal  background  information, 
including  past  experience,  and  education  and  training.  Once  the  SKAP  has  been  completed  by  the 
careerist,  the  immediate  supervisor  discusses  it  with  the  individual  employee  to  give  him/her  insights 
into  his/her  strengths  and  weaknesses. 

After  the  SKAPs  have  been  completed,  they  are  reviewed  and  annotated  by  a  Division  or  major 
command  (MACOM)  level  review  panel,  and  forwarded  to  OCE  for  review  by  an  ad  hoc  Department 
of  the  Army  (DA)  panel.  At  each  level,  individuals’  referral  eligibility  ratings  can  be  adjusted.  After 
final  review  and  adjustments  by  the  DA  panel,  the  data  on  individuals  are  ready  to  be  entered  into  the 
computer. 

At  any  time,  then,  a  computer  run  can  be  made  to  identify  all  the  personnel  in  the  Corps  who  are 
qualified  for  referral  against  a  particular  vacancy.  Before  a  recruiting  action,  the  supervisor  for  the 
vacant  position  and  the  local  personnel  office  must  indicate  applicable  referral  categories  and  SKAP  fac¬ 
tor  ratings,  if  appropriate 

Once  all  this  information  is  input  to  the  computer,  the  system  can  match  the  requirements  for  the 
vacant  position  with  the  qualifications  of  all  applicants.  This  results  in  a  list  of  potential  candidates  who 
theoretically  meet  the  prerequisites  for  the  position.  The  personnel  office  reduces  the  list  to  only  those 
who  express  an  interest  in  and  availability  for  the  vacant  position  by  responding  to  a  form  letter.  If 
more  than  10  applicants  are  on  the  list,  an  ad  hoc  panel  can  select  the  10  most  qualified  and  forward 
that  list  to  the  selecting  official.  At  this  point,  then,  the  supervisor  responsible  for  filling  the  vacant 
position  should  have  a  list  of  highly  qualified  people  from  which  to  make  a  selection. 


Reactions  to  the  SKAP  System 

Those  individuals  interviewed  who  had  experience  on  one  of  the  review  panels,  and  thus  were 
most  informed  about  SKAP,  supported  the  procedure.  The  indication  was  that  as  more  people  are 
exposed  to  the  workings  of  the  system,  SKAP  will  function  better.  But  for  the  system  to  operate  prop¬ 
erly,  the  individual  supervisors  need  to  make  fair,  realistic,  and  accurate  appraisals  of  persons  being 
rated  -  to  allow  this,  a  better  understanding  of  SKAP  is  needed.  For  example,  rotating  the  high-level 
people  who  serve  on  the  review  panels  would  expose  more  individuals  to  the  system,  workshops  and 
visits  to  sites  to  review  application  materials  and  interview  potential  candidates  would  help  promote 
more  awareness  and  understanding  of  the  system.  The  consensus,  however,  was  that  the  system’s 
operation  had  improved  considerably  since  SKAP  was  introduced,  overcoming  to  a  limited  extent  the 
extremely  inflated  ratings  which  characterized  the  first  year. 

Some  said  that  the  OCE  panel  should  not  cut  the  long  lists  generated  by  the  computer  down  to  10 
or  fewer,  since  if  the  system  is  working  properly,  all  the  engineers  on  the  list  should  be  equally 
qualified. 

A  problem  with  the  SKAP  system  seems  to  be  one  of  "management  of  forms."  The  backers  of  the 
system  apparently  feel  that  it  is  up  to  the  individual  to  insure  that  his/her  records  are  accurate  and  up 
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to  date.  However,  many  persons  seem  to  lack  adequate  information;  for  example,  when  they  fill  out 
SKAP  forms,  they  fail  to  realize  the  importance  of  the  details  of  certain  numbers  and  definitions  on  the 
forms.  Some  people  did  not  become  acutely  aware  of  the  importance  of  such  details  until  they  nearly 
missed  out  (or  in  fact  did  miss  out)  on  a  promotion  because  of  erroneous  data  in  the  system.  Again, 
this  is  probably  largely  a  problem  of  educating  people  about  the  system.  However,  there  were  some 
indications  that  the  procedures  for  updating  and  correcting  information  in  the  system  might  not  be 
efficient.  For  instance,  the  personnel  offices  could  take  a  more  active  role  in  keeping  the  personnel  files 
updated.  If  computer  personnel  cannot  do  the  necessary  quality  control  checking,  some  procedure  is 
needed  to  provide  the  personnel  office  and  the  individual  employee  immediate  feedback  on  updates. 

From  the  information  gathered  through  interviews,  a  variety  of  suggestions  for  changes  which 
might  improve  the  operation  of  the  SKAP  system  have  been  formulated.  Some  of  these  suggestions 
came  directly  from  the  interviewees;  most  were  formulated  from  the  interviewers'  overall  impressions. 

1 .  Several  people  said  there  should  be  more  human  input  to  the  system  and  not  total  reliance  on 
the  machine.  One  interviewee  offered  a  specific  idea:  "When  the  various  ad  hoc  committees  go  over 
the  list  of  SKAP-recommended  candidates,  they  should  be  able  to  add  the  names  of  qualified  people  to 
the  list  as  well  as  take  people  off."  A  further  suggestion,  which  was  mentioned  by  a  few  people,  was  to 
return  to  filling  positions  by  an  ad  hoc  committee  which  discusses  the  available  candidates  and  produces 
a  list  of  those  thought  highly  qualified.  (However,  such  procedures  could  introduce  to  the  system  less 
objectivity  and  a  lack  of  control.)  This  point  of  view  illustrates  how  much  information  about  the  SKAP 
system  is  needed. 

2.  Apparently  there  has  been  a  problem  with  people  turning  down  jobs  in  a  location  they  had 
indicated  was  acceptable.  The  possibility  of  taking  punitive  action  against  individuals  who  turn  down 
such  job  offers  has  been  considered.  One  supervisor  said,  "If  you  aren’t  sure  you  would  take  such  a 
job,  don’t  mark  it  on  your  SKAP.  The  procedure  can  take  3  to  6  months  to  get  to  the  point  of  offering 
someone  a  position  once  it  becomes  vacant;  therefore,  I  don’t  want  to  get  to  that  point  and  have  to 
start  all  over."  Most  employees,  however,  are  very  disturbed  by  this  attitude.  They  feel  they  have  the 
right  to  consider  positions  as  they  come  up  and  should  be  able  to  mark  location  X  even  if  they  would 
only  consider  very  few  positions  there.  The  employee  can  be  fairly  restrictive  on  the  SKAP  about  what 
jobs  he/she  would  accept,  but  still  might  consider  a  wider  range  of  positions  at  one  location  than  at 
another. 

3.  One  objective  of  the  SKAP  system  was  to  increase  effective  discussion  between  supervisor  and 
subordinate  concerning  job  duties,  behavior,  and  career  development.  While  this  goal  may  be  met  with 
varying  degrees  of  success,  a  frequent  complaint  was  that  it  does  not  include  performance  feedback  or 
appraisals.  To  promote  such  discussion  and  increase  the  objectivity  of  SKAP’s  information,  one  inter¬ 
viewee  suggested  a  dual  performance  appraisal  involving  the  supervisor  and  a  personnel  professional. 
He  knew  of  a  large  industrial  firm  which  uses  this  strategy.  The  outside  professional  introduced  more 
objectivity  into  the  procedure  and  was  adept  at  counseling  and  career  development. 

4.  Another  suggestion  was  that  independent  performance  ratings  be  obtained  from  first-  and 
second-level  supervisors,  who  would  then  discuss  and  resolve  any  differences  in  these  ratings  before  the 
results  were  discussed  with  the  employee.  Typically,  such  ratings  have  a  somewhat  modest  level  of  reli¬ 
ability,  approximately  in  the  .60s.  Nonetheless,  the  ratings  could  permit  the  accumulation  of  useful 
data  for  research  on  the  reliability  of  ratings  within  the  Corps,  and  could  provide  a  valuable  contribu¬ 
tion  to  a  training  program  on  performance  appraisals  and  personnel  evaluation.  Furthermore,  such  pro¬ 
cedures  would  tend  to  produce  more  effective  discussions  between  supervisors  and  more  objective 
evaluations  because  differences  would  be  obvious  and  would  have  to  be  resolved. 

5.  There  has  been  a  problem  with  supervisors  rating  nearly  all  people  very  high  (M)  on  nearly  all 
SKAP  traits.  For  the  M  category,  the  descriptive  statements  of  the  various  elements  in  Form  4428 
specify  near  perfection,  and  not  many  people  truly  function  at  that  level  on  more  than  a  few  of  the  ele¬ 
ments.  The  descriptors  were  written  to  present  such  a  level  of  functioning  that  only  a  few  would  be 
rated  highest  (M),  with  the  rest  dispersed  across  the  other  two  levels  (S  and  A).  Workshops  or  other 
informative  devices  should  instill  in  supervisors  the  idea  that  employees  ought  to  be  distributed  across 
the  three  levels  of  performance  for  the  various  elements.  Supervisors  apparently  believe  now  that  any¬ 
thing  less  than  the  highest  rating  on  any  element  is  a  condemnation. 


6.  One  suggestion  having  real  merit  was  to  present,  as  part  of  the  SKAP  printout,  a  given 
supervisor’s  average  ratings  of  his/her  subordinates.  This  would  help  identify  the  very  lenient  supervi¬ 
sor  who  has  employees,  according  to  one  interviewee,  "that  walk  on  water  without  getting  their  feet 
wet.”  These  data,  if  not  already  in  computer  storage ,  could  be  obtained  with  minimal  difficulty  in  any 
revision  of  the  SKAP  system. 

7.  An  alternate  method  of  dealing  with  ratings  that  are  too  lenient  (or  too  strict)  would  be  to 
have  the  supervisor  enter  on  the  SKAP  record  the  employee's  three  areas  of  greatest  strength  and  the 
three  areas  in  which  the  employee  has  the  greatest  opportunity  for  improvement.  This  technique  would 
force  supervisors  to  think  about  the  different  levels  of  performance  of  individual  employees,  and  the 
results  could  be  keyed  into  the  computer  when  the  names  of  qualified  people  are  screened  for  the  list  of 
potential  candidates  for  open  positions. 

The  SKAP  procedures  have  evidently  improved  the  degree  of  objectivity  and  documentation 
involved  in  promotions,  though  there  is  variability  in  the  system’s  use.  A  number  of  individuals  indi¬ 
cated  that  they  used  the  information  from  the  system  to  evaluate  patterns  of  high  and  low  ratings  and 
interpretations  of  employees’  written  work.  On  the  other  hand,  some  supervisors  indicated  that  they 
merely  looked  at  the  employee’s  absolute  number  of  M  ratings  and  publications. 

The  result  is  that  too  many  people  still  distrust  the  SKAP  procedures  and  want  either  to  make 
their  own  decisions  on  the  basis  of  their  authority  or  to  seek  out  new  information  from  their  own 
sources.  One  GS- 1 5  commented,  "1  put  very  little  emphasis  on  performance  appraisals.  They  are  sim¬ 
ply  too  arbitrary.  The  flick  of  a  pen  makes  a  substantial  difference  in  a  person’s  career.  The  perfor¬ 
mance  appraisal  is  a  necessary  evil.  It  is  much  easier  to  give  higher  ratings  than  to  try  to  defend  and 
work  through  lower  ratings.  In  addition,  the  SKAP  system  tends  to  carry  a  loser."  When  this  supervisor 
fills  a  position  he  tries  to  pick  someone  he  knows  and  thus  avoid  any  problems.  This  is  a  difficulty 
which  is  very  hard  to  overcome.  It  is  simply  human  nature  to  pick  a  known  quantity  and  avoid  prob¬ 
lems  rather  than  to  use  a  system  which  can  introduce  unknowns  to  the  decision  making  process. 

Reactions  from  the  field  indicate  that  the  SKAP  system  is  not  well  understood.  The  primary 
misunderstanding  is  this:  the  SKAP  system  is  not  a  performance  appraisal  system  but  is  considered  just 
that  by  many  of  its  users,  who  are  also  uncertain  about  how  the  paneling  process  works,  and  about  how 
lists  of  candidates  are  generated. 


Selection  Strategies  and  Ideas 

Interviewees  were  concerned  with  maintaining  flexibility  and  local  autonomy  in  use  of  all  selection 
procedures.  This  concern  applied  to  the  SKAP  or  any  other  system  that  might  result  from  the  current 
study.  There  was  a  general  sense  that  someone  in  the  District  should  get  the  position  if  he/she  were 
equally  or  better  qualified  than  outside  candidates.  Generally  associated  with  this  was  the  belief  that  "it 
is  highly  unlikely  that  in  a  District  of  this  size  there  is  not  someone  highly  qualified  for  any  particular 
position."  This  naturally  aids  continuity  by  moving  people  up  through  the  ranks  and  is  also  an  inherent 
part  of  morale  in  most  healthy  organizations;  i.e.,  employees  feel  "there  is  a  future  around  here."  The 
individual  supervisor’s  concern  about  being  able  to  do  what  he/she  wants  -  without  the  constraints  of 
general  policies,  personnel  regulations,  stopper  lists,  and  Civfl  Service  requirements  -  was  much 
stronger  than  the  concern  about  reducing  inbreeding.  Even  though  most  hiring  officials  talk  about  how 
desirable  cross-training  and  mobility  are,  they  generally  want  to  fill  a  vacancy  with  people  they  know 
have  proven  ability.  And  they  do  not  want  someone  who  is  an  "unknown  quantity"  moved  into  a  posi¬ 
tion.  So  even  though  the  system  says  a  person  is  qualified,  some  supervisors  are  not  prepared  to 
believe  it.  Therefore,  most  managerial  jobs  apparently  are  filled  by  a  person  who  worked  under  that 
position;  if  that  does  not  work,  hiring  officials  are  willing  to  look  elsewhere  in  the  District;  and  if  that 
does  not  work  ("highly  unlikely,"  according  to  one  respondent),  to  look  beyond  the  District. 

This  is  obviously  a  very  complex  situation  since  most  individuals  are  only  willing  to  give  lip  ser¬ 
vice  to  cross-training  (including  mobility)  and  the  reduction  of  inbreeding.  Some  strongly  worded 
directives,  personnel  policies,  or  selling  programs  will  have  to  be  established  to  overcome  reluctance 
about  bringing  individuals  in  from  other  Districts.  One  guideline  seems  clear  if  possible,  some  flexi¬ 
bility  and  autonomy  in  the  use  of  selection  tools  --  SKAP,  for  example  -  should  be  provided  at  the 
local  level.  Otherwise,  supervisors  probably  will  not  accept  the  tools  without  an  extensive  training 
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program  in  which  the  instruments  are  thoroughly  sold  so  their  benefits  are  clearly  recognized.  In  other 
words,  the  tools  might  not  be  used  if  they  do  not  provide  the  option  of  selecting  the  candidate  supervi¬ 
sors  want.  These  comments  about  local  autonomy  and  flexibility  should  be  borne  in  mind  as  alternative 
selection  procedures  are  described  below. 

Interviewing  Procedures 

Essentially  everyone  felt  that  an  interview  with  final  applicants  not  known  to  the  selecting  supervi¬ 
sor  was  necessary.  A  number  of  individuals  were  questioned  about  their  interviewing  techniques, 
which  varied  substantially.  Only  a  small  percentage  of  individuals  used  any  systematic  interviewing 
strategy.  Since  the  number  of  people  selected  for  high  level  management  positions  is  relatively  small, 
few  supervisors  gain  much  experience  in  filling  such  positions.  Moreover,  developing  interviewing  and 
other  selection  skills  is  difficult  because  of  the  large  time  span  between  vacancies.  Some  sort  of  sys¬ 
tematic  interview  could  be  of  substantial  benefit.  However,  since  so  many  unstructured  procedures 
have  been  employed,  the  interview  is  the  most  widely  used  ineffective  selection  procedure  available  - 
and  that  assessment  is  probably  especially  applicable  to  the  Corps.  One  individual  reported  that  the 
interview  for  the  selection  of  a  high-level  supervisor  is  sometimes  a  very  brief  affair,  as  short  as  20 
minutes.  Others  reported  using  no  interview  guide  or  pre-identified  questions  for  the  interview;  they 
wanted  a  chance  to  "react  spontaneously"  to  the  applicant  and  his/her  experience.  While  a  portion  of 
any  interview  could  —  and  perhaps  should  —  be  unstructured,  an  effective  interview  should  contain  key 
questions  established  beforehand  --  that  is,  those  questions  concerning  job  requirements  and  experi¬ 
ence. 

Peer  Nominations 

The  possibility  of  using  peer  nominations  as  indicators  of  performance  was  brought  up  by  the 
investigators  and  discussed  in  the  interviews.  The  reactions  varied  from  uncertain  acceptance  to  strong 
support.  Most  managers  from  the  GS-13  through  -15  levels  would  have  worthwhile  input  about  the 
managerial  performance  and  potential  of  at  least  30  percent  of  the  other  13s  and  15s  in  the  District. 
Thus,  a  peer  nomination  and  evaluation  procedure  appears  very  feasible  for  Corps  use.  In  fact,  such  a 
procedure  is  now  used  by  Standard  Oil  of  New  Jersey  and  other  industrial  firms.  There  is  associated 
software  that  can  be  used  to  synthesize  the  data  from  a  number  of  sources  and  provide  a  computer  out¬ 
put  of  the  results.  Although  the  exact  details  of  the  procedure  would  have  to  be  adapted  to  Corps 
requirements,  the  general  system  would  work  something  like  this: 

Three  meetings  are  scheduled  in  each  District  annually  so  that  all  supervisors  (GS-13  through 
-15)  can  attend  at  least  one.  At  each  meeting  those  attending  are  given  a  deck  of  computer  cards  with 
the  names  of  all  13s  through  15s  printed  one  per  card.  The  supervisor  then  sorts  the  cards  into  two 
stacks  -  those  in  or  not  in  his/her  functional  division.  Each  of  the  two  decks  is  then  sorted  further  so 
that  those  individuals  not  known  well  by  the  supervisor  are  set  aside.  At  this  point,  the  supervisor  has 
three  stacks  of  cards  -  those  in  his/her  functional  division  well  enough  known  to  be  ranked,  those  not 
in  his/her  functional  division  but  well  enough  known  to  be  ranked,  and  those  not  well  enough  known 
to  be  ranked.  The  supervisor  then  sets  aside  the  last  group  and  proceeds  to  rank  those  in  the  other  two 
decks  on  overall  management  capability  as  defined  for  the  supervisor  on  a  handout  sheet.  Each  of  the 
two  decks  is  arranged  in  rank  order  and  secured.  This  entire  procedure  can  be  done  quite  rapidly  - 
especially  after  individuals  become  familiar  with  it.  The  cards  are  then  fed  into  the  computer  program, 
which  analyzes  the  data  by  maximizing  the  agreement  between  observers  and  developing  a  composite 
rating  for  each  individual  within  and  outside  each  division. 

The  advantages  of  this  procedure  are  that  it  introduces  additional  objectivity  into  the  system  and 
does  not  rely  strictly  on  the  leniency  or  the  accuracy  of  the  immediate  supervisor.  Since  the  actual 
rankings  by  participating  nominators  are  anonymous,  there  is  additional  objectivity  in  the  results.  And 
because  the  individuals  doing  the  ranking  all  see  the  persons  being  ranked  from  different  perspectives, 
there  is  considerably  more  breadth  of  input  into  the  process  than  any  one  reader  or  ranker  could  pro¬ 
vide.  The  combined  peer  nomination  data  can  be  used  as  another  input  to  (not  the  determinant  of)  all 
selection  and  promotion  decisions.  Conceivably,  such  a  procedure  also  could  be  applicable  to  other 
grades  of  personnel. 


The  Ad  Hoc  Process 

With  the  ad  hoc  process,  used  at  two  of  the  locations  visited,  an  ad  hoc  group  --  usually  composed 
of  three  individuals  from  outside  the  immediate  chain  of  command  for  the  open  position  --  evaluates 
the  characteristics  of  available  candidates  for  the  requirements  of  the  vacant  position  (for  GS  grades  14 
and  under).  Typically,  the  first  or  second  level  supervisor  for  the  open  position  establishes  weights  for 
various  SKAP  elements  or  other  categories  of  job  requirements.  The  group  member  then  rates  appli¬ 
cants  on  each  of  the  categories,  and  the  sum  of  the  ratings  multiplied  by  the  weights  assigned  to  each 
category  results  in  a  total  score  for  the  applicant.  The  results  are  then  reviewed  and  a  list  of  the  three 
to  five  highest  scoring  applicants  is  given  to  the  selecting  supervisor.  Since  the  ratings  for  each  appli¬ 
cant  are  based  on  the  information  submitted,  individuals  from  a  selecting  location  usually  had  more 
information  available  in  their  folders  and  therefore  had  an  advantage  over  applicants  from  the  outside. 

The  ad  hoc  process  can  remove  pressures  from  the  immediate  supervisor  and  allow  a  relatively 
objective  group  of  supervisors  to  select  the  final  candidates  for  a  position.  Overall,  then,  the  process 
appears  to  have  some  important  advantages.  Most  importantly,  it  increases  the  objectivity  of  the  selec¬ 
tion  decision. 

However,  the  ad  hoc  process  might  be  improved  in  a  number  of  ways.  The  process  takes  a  sub¬ 
stantial  amount  of  time  because  group  members  have  to  study  personnel  folders  and  have  extensive 
discussions.  The  process  might  be  made  more  efficient  with  brief  summaries  prepared  by  the  personnel 
office  or  by  the  applicant.  These  summaries  would  contain  the  key  aspects  of  each  applicant's  back¬ 
ground  and  experience  relative  to  the  position  and  could  be  used  as  a  primary  source  of  information  for 
the  evaluations.  The  summaries  should  be  identified  only  with  a  social  security  number  to  increase  the 
objectivity  of  the  procedure.  Another  point  of  potential  concern  was  the  way  categories  for  job  require¬ 
ments  were  established.  There  was  some  indication  that  this  was  an  arbitrary  and  hasty  procedure, 
completed  without  careful  consideration  of  the  key  requirements  for  successful  performance  on  the  job 
If  applicants  are  selected  on  loosely  defined,  partially  irrevelant  characteristics,  the  whole  procedure  can¬ 
not  efficiently  match  the  best  individual  to  the  real  job  requirements. 

One  of  the  locations  visited  also  used  an  ad  hoc  procedure  for  interviewing  final  candidates;  indi¬ 
viduals  were  interviewed  by  a  committee  using  either  structured  or  unstructured  interview  questions. 
The  panel  arrived  at  a  consensus  and  presented  the  final  ranking  to  the  selection  supervisor.  The 
recommendations  of  the  panel  were  essentially  binding  on  the  selecting  supervisor,  who  had  to  pick  the 
top  person  on  the  list  unless  there  was  a  substantial  reason  for  not  doing  so.  Although  this  process 
helped  reduce  the  emphasis  on  a  single  interviewer’s  results,  such  use  of  the  interview  was  inefficient, 
in  contrast  with  what  might  have  been  accomplished  by  a  well  planned  interview  procedure.  Nonethe¬ 
less,  because  of  its  breadth  of  impact  from  a  variety  of  sources,  the  process  appears  to  have  a  marked 
advantage  over  interviewing  by  a  single  supervisor. 


Characteristics  of  a  Good  Manager 

In  a  number  of  the  interviews,  the  participants  were  asked  to  list  the  qualities  of  an  effective 
Corps  manager.  An  alternate  procedure  was  to  ask  for  a  description  of  the  most  and  least  effective 
manager  the  participant  had  ever  known.  These  questions  generated  a  long  list  of  characteristics  from 
which  some  key  behaviors  and  attributes  of  effective  managers  could  be  identified.  Such  information 
typically  is  difficult  to  work  with  because  it  usually  consists  of  a  long  list  of  laudatory  adjectives  which 
are  difficult  to  translate  into  practice.  On  the  other  hand,  the  responses  do  produce  a  number  of  clues 
that  can  be  related  to  various  selection  strategies.  Furthermore,  an  overall  view  of  effective  managers' 
attributes,  as  seen  by  people  in  the  field,  was  important  to  the  study. 

Table  1  summarizes  the  various  responses  and  shows  that  characteristics  were  organized  in  three 
overall  categories;  managerial  skills,  personal  characteristics,  and  technical  background.  The  overall 
image  appears  to  be  that  of  someone  who  is  perceptive  and  alert,  displaying  energy  and  commitment 
The  effective  manager  is  seen  as  one  who  builds  people  in  the  long  run  instead  of  using  them  in  the 
short  run,  and  thus  must  be  primarily  oriented  toward  people  rather  than  facts  or  projects.  Such  a  per¬ 
son  is  involved  in  a  wide  variety  of  activities  and  looks  for  opportunities  to  challenge  his/her  subordi¬ 
nates. 
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While  Table  1  presents  information  in  general  categories,  many  of  the  more  specific  characteristics 
mentioned  have  been  incorporated  into  the  interview  guide  discussed  in  Chapter  7.  The  information 
obtained  from  this  part  of  the  interviews  could  also  supplement  the  biographical  items  discussed  in 
Chapter  5. 


Concerns  of  the  Interviewees 

Several  individuals  voiced  concerns  about  this  study;  that  is,  what  was  its  direction  and  what  pro¬ 
ducts  might  result  from  it.  One  concern  was  that  the  study  should  not  result  in  an  early  identification 
procedure  that  would  classify  personnel  as  "peons"  or  "fair-haired  boys"  who  are  destined  to  move  up  to 
high-level  positions.  Individuals  felt  that  such  a  procedure  removes  the  incentive  to  produce  and  would 
not  be  a  fair  process  continually  sensitive  to  individual  performance  and  potential. 

Another  concern  was  that  the  study  might  produce  more  paperwork  for  managers  and  employees. 
People  felt  they  presently  had  to  spend  too  much  time  on  paperwork  of  no  real  value.  They  were  con¬ 
cerned  that  a  long  and  complicated  rating  form  for  everyone  would  add  to  their  workload. 


Table  1 

Characteristics  of  Outstanding  Managers 


Managerial: 

Communication  Skills  -  oral  (with  employees,  clients,  contractors, 
etc. )  and  written 

Broad  Perspective  -  seeks  information  from  as  many  sources  as  possible 

Results  Orientation  -  interest  in  high  standards,  not  techniques 

Willingness  to  Delegate  Responsibility  and  Back  Up  Employees 

Ability  to  Motivate  and  Stimulate  People 

Concern  about  Career  Development  of  Employees 

Effective  Analytic  and  Decision-Making  Skills 

Ability  to  Accept  Responsibility  and  Back  Decisions  from  Above 


Personal 

Integrity-Honesty-Fairness  --  respected  by  coworkers  and  subordinates 
Dedication-Involvemem-Responsiveness  -  evidence  of  high  commitment 
Interpersonal  Skill-Sensitivity-Tact 
Motivation  to  Achieve  -  compulsive  worker 


Technical 

Breadth  of  Experience  -  including  field,  District.  Division,  OCE, 
and  perhaps  other  organizations 


Competence  in  Specialty  and  Related  Areas 


Others  were  concerned  about  what  seemed  to  be  an  additional  layer  of  constraints  which  would 
inhibit  their  freedom  to  act  as  they  thought  best  for  their  position  and  the  Corps.  Constraints  that  are 
applied  universally  would  not  be  well  received  in  the  field  unless  their  need  were  thoroughly  explained. 

One  individual’s  thoughts  summarize  the  concerns  of  the  interviewees: 

1.  "Be  careful  that  all  forms  and  procedures  do  not  become  dehumanizing." 

2.  “Above  all,  the  system  must  be  fair  and  honest.” 


Role  of  the  Interview  Data 

After  the  months  of  interviewing  and  discussions,  the  information  obtained  was  organized  into  a 
general  framework  which  guided  the  following  2  years  of  research.  Figure  1  illustrates  the  conceptual 
framework:  for  any  selection  or  promotion,  an  opening  has  to  exist;  to  fill  that  open  position,  the 
requirements  of  the  position  must  be  determined  so  that  candidates  can  be  screened  as  the  selection 
process  begins,  all  these  steps  take  place  within  the  organization  where  the  opening  exists  and  the 
selected  individual  has  to  perform. 


JOB  OPENING 

_ 1 

DEFINITION  OF  JOB 

REQUIREMENTS 

_ 1 

- 

SELECTION  PROCESS 

JOB  PERFORMANCE  PERSONAL  TRAITS 

INFORMATION 

! 

ORGANIZATIONAL 

TRAITS 

Figure  I.  Conceptual  framework. 
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3  JOB  ACTIVITIES  DESCRIPTION  QUESTIONNAIRE 


Development  of  the  JAD  Questionnaire 

The  Job  Activities  Description  (JAD)  Questionnaire  was  developed  to  gain  information  from 
engineers  at  the  GS-l!  level  and  higher  concerning  important  job .  activities  within  the  Corps  of 
Engineers.1  The  rationale  for  the  JAD  was  that  engineering  jobs  at  different  levels  emphasize  different 
activities,  which  also  vary  in  importance  because  the  duties  and  responsibilities  of  the  different  levels 
vary.  By  focusing  on  the  activities  of  engineers  at  each  level  within  the  Corps,  this  study  could  exam¬ 
ine  the  job  skills  important  for  an  engineer  to  function  competently  at  the  higher  levels.  Thus,  the 
JAD  analyzed  management  policies  that  were  generally  characteristic  of  key  positions  within  the  Corps. 

Two  possible  uses  of  the  JAD  were  envisioned  as  the  instrument  was  designed: 

1.  The  instrument,  or  a  revision  of  it,  potentially  could  be  completed  either  by  the  incumbent  of 
a  position  or  the  supervisor  of  a  position  that  was  soon  to  be  vacant.  The  information  obtained  from 
either  or  both  of  these  sources  would  provide  specific  data  about  key  job  requirements  for  the  position. 
This  information,  in  cot\junction  with  data  from  other  sources,  could  be  used  during  the  selection  pro¬ 
cess  to  help  match  the  characteristics  of  the  individual  to  job  requirements. 

2.  Certain  trends  in  JAD  results  showed  that  the  duties  of  engineers  at  the  GS-l  I  and  GS-l 2  lev¬ 
els  are  different  from  those  of  engineers  at  GS-l 3  through  -15.  So  if  certain  job  characteristics  or 
duties  were  related  to  a  particular  GS  level,  these  activities  could  be  evaluated  -  in  interviews,  revised 
research  instruments,  personnel  forms  —  as  part  of  the  general  selection  process  for  all  managerial  posi¬ 
tions  at  that  grade.  Thus,  candidates  would  be  assessed  for  their  experience  in,  or  knowledge  of,  the 
activities  important  to  success  in  management  jobs  at  a  given  GS  level. 

For  the  preliminary  work  on  the  JAD,  a  pilot  questionnaire  was  designed  and  then  administered 
to  394  engineers  in  GS  grades  11  through  16  at  three  Corps  District  offices.  Development  both  of 
categories  describing  job  responsibilities  and  of  specific  activities  within  each  category  was  not  to  imply 
that  engineers  cannot  or  do  not  contribute  to  the  effectiveness  of  the  organization  in  other  ways 
Rather,  the  JAD’s  general,  discrete  categories  were  constructed  simply  to  have  some  value  in  selecting, 
training,  and  evaluating  management.  Results  obtained  from  394  engineers  indicated  strong  relation¬ 
ships  between  GS  level  and  the  rated  importance  of  job  activities.  Higher  level  managers,  for  instance, 
spent  more  time  in  activities  related  to  training  and  motivating  personnel  than  did  GS-l  Is,  and  reported 
that  these  activities  were  of  much  greater  importance  to  their  job.  However,  results  from  this  initial 
design  and  testing  of  the  JAD  were  somewhat  limited  because  only  a  small  number  of  higher  grade  per¬ 
sonnel  (GS-l 5  and  -16)  were  included. 

Nevertheless,  the  pilot  study  pointed  toward  some  important  design  revisions.  For  the  JAD 
described  in  this  report,  the  time  scales  were  revised  and  a  somewhat  different  set  of  activities  within 
categories  was  used. 


Objectives  of  JAD  Revision 

The  general  goal  of  the  research  documented  in  this  chapter  was  to  determine  the  refined  JAD’s 
usefulness  for  supplementing  current  personnel  selection  procedures.  The  JAD  would  both  define 
activities  important  for  success  in  higher  level  jobs,  and  indicate  their  differences  from  those  in  jobs  at 
the  journeyman  level  of  GS-l  1.  Specifically,  the  objectives  of  the  study  were  to. 

1.  Ascertain  the  differences  in  time  spent  on  selected  job  activity  categories  by  different  levels  of 
management. 


1  For  a  complete  discussion  of  JAD  development,  see  R.  L  Ellison,  C.  Abe.  D  G.  Fox,  and  W  D  Veneklasen.  TV  Job  Activi¬ 
ties  Description  Questionnaire :  An  Analysis  of  Time  Spent  on  and  Importance  of  Managerial  Duties,  Interim  Report  F- 
I57/ADA074I75  (U  S  Army  Construction  F.ngineering  Research  Laboratory  ICERLl.  September  1079) 


2.  Determine  the  relative  importance  of  selected  job  activities  across  different  levels  of  manage¬ 
ment. 

3.  Examine  various  job  classifications  for  any  differences  in  selected  job  activities. 


Procedure 

Form  Development 

The  JAD  benefited  substantially  from  previous  research,  having  been  constructed  on  the  basis  of 
work  reported  by  Hemphill  and  Mahoney.  These  studies  and  others,  such  as  the  early  work  by 
Flanagan  on  critical  incidents,  focused  principally  on  the  many  activities  of  managers  and  on  how  jobs 
differ  within  and  across  organizations.2 3  Obviously,  a  number  of  different,  complex  activities  are  charac¬ 
teristic  of  managerial  positions,  and  these  early  studies  documented  this  complexity.  Such  studies  have 
important  implications  for  training  and  for  assessing  job  performance  since  they  indicate  that  a  great 
variety  of  skills  and  abilities  is  required  for  competence  as  a  manager.  These  studies,  however,  did  not 
indicate  how  managerial  positions  should  be  evaluated  and  did  not  describe  the  nature  of  managerial 
positions  at  different  levels  of  the  organization.  Also,  no  information  was  available  on  how  Corps 
managers  may  be  different  from  managers  in  other  organizations. 

For  the  pilot  form  of  the  JAD,  10  categories  of  job  behaviors  were  selected  -  on  the  basis  of  the 
earlier  research  and  interview  results  -  as  relevant  to  Corps  activities.  Within  each  of  these  10 
categories,  at  least  three  specific  job  behaviors  were  included  to  pinpoint  the  actual  nature  of  what 
Corps  managers  do.  Managers  at  different  levels  evaluated  their  jobs  on  each  of  the  behaviors  by  indi¬ 
cating  both  the  amount  of  time  spent  on  each  activity  and  the  importance  of  each  activity  to  success  in 
their  particular  positions.  The  results  indicated  that  very  few  differences  existed  in  time  spent  across 
different  levels  of  management.  Such  findings  have  been  typical  in  other  research  studies  as  well.4  Of 
more  value  were  perceptions  by  different  levels  of  management  about  the  relative  importance  of 
specific  job  activities. 

This  pilot  work  had  a  limited  sample  of  higher  level  management  personnel,  particularly  those  at 
GS  levels  15  and  16.  In  addition,  further  refinements  were  made  in  the  design  of  both  the  categories 
and  the  specific  job  activities  presented  on  the  JAD  form.  These  changes  were  made  to  improve  the 
relationship  between  the  categories  and  the  different  managerial  levels,  and  to  improve  the  sensitivity 
of  the  instrument  to  these  different  levels. 

One  concern  in  the  design  of  the  JAD  was  the  provision  of  appropriate  items  so  that  lower  GS 
level  personnel  could  describe  their  positions  by  the  amount  of  time  spent  on  and  the  importance  of  the 
particular  job  activity.  Including  those  lower  level  job  activities,  in  some  cases,  was  counter  to  the  pur¬ 
pose  of  the  questionnaire  -  i.e.,  defining  the  job  activities  of  higher  level  managers.  However,  exclud¬ 
ing  the  categories  through  which  lower  level  personnel  could  describe  their  positions  would  have  been 
detrimental  to  the  study  and  to  the  form.  Thus,  a  number  of  job  activities  which  were  more  charac¬ 
teristic  of  lower  level  personnel  were  written  to  provide  an  opportunity  to  describe  these  positions  and 
thus  detect  differences  between  these  and  the  higher  level  positions. 

Figures  2  and  3  present  the  revised  form  of  JAD  used  in  the  current  study. 


2  J  K.  Hemphill,  ed.,  The  Engineering  Study  (Educational  Testing  Service,  1963);  ).  K  Hemphill,  "Job  Descriptions  for  Execu¬ 
tives,"  Harvard  Business  Review,  Vol  37  (1959),  pp  55-67;  T.  A.  Mahoney,  Criteria  of  Organizational  Effectiveness,  Mimeographed 
report  (University  of  Minnesota,  Industrial  Relations  Center,  1966). 

’  J.  C.  Flanagan,  "The  Critical  Incident  Technique."  Psychological  Bulletin,  Vol  51  (1954),  pp  327-358 

4  J  K.  Hemphill,  ed..  The  Engineering  Study  (Educational  Testing  Service,  1963);  J.  K.  Hemphill.  "Job  Descriptions  for  Execu¬ 
tives,"  Harvard  Business  Review,  Vol  37  (1959),  pp  55-67. 
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The  Job  Activities  Description  will  provide  information  about  the  kinds  of  activities 
that  are  required  by  certain  positions  within  the  Corps.  The  data  will  be  very  valuable 
in  helping  to  define  and  understand  the  kinds  of  activities  that  are  carried  out  by  key 
Corps  personnel.  Ultimately,  the  information  from  this  form  will  contribute  to  the  future 
selection  and  placement  of  personnel  into  key  managerial  positions.  Your  cooperation  in 
collating  the  form  is  greatly  appreciated. 

All  data  will  be  treated  confidentially  and  anonymously.  The  information  you  provide  in 
the  section  below  will  aid  in  the  effective  use  of  the  data  provided  on  the  back  of  this 

page. 


1.  G.  S.  level _ 

2.  Years  of  professional  experience _ 

3.  Current  job  title _ 

Please  indicate  if  you  have  had  two  or  more  years  of  experience  in  each  of 
the  following  areas  by  marking  yes  or  no  on  the  line  following  each  area. 
Please  circle  the  number  of  the  area  in  which  you  are  presently  working. 

4.  Civil  Works  _ 

5.  Military  Construction  _ 

o.  Engineering  _ 

Planning  _ 

8.  Construction  _ 

9.  Operations  _ 

10.  Other 


The  back  of  this  page  is  a  form  for  providing  information  about  the  kinds  of  activities 
which  are  important  for  a  person  in  your  job.  For  each  activity  listed,  please: 

(a)  use  the  scale  on  the  left  to  indicate  the  approximate  percent  of  time  that 
you  spend  in  each  major  category  of  activity  printed  in  capital  letters.  For  example, 
if  you  spend  about  7%  of  your  time  in  activities  which  could  be  classified  under  INTERACT 
WITH  OTHER  ORGANIZATIONS,  you  should  write  7  in  the  blank  to  the  left  of  that  category. 
These  percentages  will  not  total  100,  since  some  activities  overlap  and  others  are 
omitted. 


(b)  use  the  scale  on  the  right  to  indicate  how  important  it  is  for  a  person  in 
your  job  to  perform  adequately  in  the  specific  activities  within  each  category.  For 
example,  if  "Organize  work  schedules"  is  a  crucial  job  activity,  write  a  5  in  the  space 
to  show  this.  When  rating  the  activities,  disregard  the  percent  of  time  you  have  just 
indicated  and  how  hard  it  may  be  to  do  the  activity.  Please  rate  all  of  the  specific 
activities . 


Figure  2.  Job  Activities  Description:  page  1. 


IMPORTANCE 


PERCENT 
OF  TIME 


REVIEW  AND  EVALUATE.  PROJECTS 

Monitor  project  progress . 

Evaluate  productivity  measures  and  systems. 
Review  cost  accounting  records . 

ALLOCATE  RESOURCES 

Budget  program  expenditures  . 

Allocate  resources  to  projects  and  programs 
Prepare  cost  estimates  for  a  project.  .  .  . 


Crucial 

Quite  Important 

Moderate  laportance 

Some  Importance 

Unimportant 

i  I 

i  :  s  a  s 


TRAIN  AND  MOTIVATE  PERSONNEL 

Analyte  training  and  developmental  needs  of  subordinates 

Guide  personnel  assignments  and  procedures . 

Provide  for  personnel  development  and  recognition  .  .  . 


INITIATE  IMPROVEMENTS  IN  METHODS  AND  OPERATIONS 

Develop  and/or  implement  new  methods  of  operation 
Plan  and  initiate  new  organizational  procedures  . 
Design  new  management  procedures . 

USE  KNOWLEDCE  AND  SKILLS  IN  A  TECHNICAL  CAPACITY 

Solve  various  engineering  and  scientific  problems 
Prioritize  system  requirements  and  objectives  .  . 
Act  as  consultant  for  solving  technical  problems. 


INTERPERSONAL  INTERACTION 

Help  others  resolve  differences  . 

Stimulate  cooperation  among  employees  . 

Provide  feedback  to  subordinates  about  job  performance . 

INTERACT  WITH  OTHER  ORGANIZATIONS 

Negotiate  with  outside  organitations  . 

Develop  cooperative  relationships  with  clients,  interest  groups,  etc. 
Keep  up  with  current  events  through  contacts  with  outsiders  . 

SCHEDULE  WORK  OF  ORGANIZATIONAL  COMPONENTS 

Organize  activities  of  others  to  respond  to  problems . 

Organize  work  schedules  . 

Establish  project  priorites  . 

LONG-RANGE  PLANNING  AND  CONTROL 

Participate  in  long-range  planning  meetings  . 

Investigate  possible  future  developments  and  response  options  .  .  .  . 
Develop  long-range  priorities  .  . 


Figure  3.  Job  Activities  Description:  page  2. 
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The  measurement  of  time  spent  was  requested  only  for  the  most  general  categorizations  of  job 
activities.  In  the  first  JAD  study  and  other  research,  estimates  of  the  percentage  of  time  spent  on 
specific  job  activities  were  so  small  that  accurate  measures  were  difficult  to  obtain;  therefore,  it  was 
thought  greater  abstraction  would  provide  more  valuable  data  on  time  spent.5 

On  the  other  hand,  previous  research  with  JAD  had  demonstrated  that  the  more  specific  the  state¬ 
ment  of  the  actual  job  activity,  the  more  likely  that  the  statement  would  differentiate  between  various 
levels  of  management  performance.  Thus,  the  specific  job  activities  were  rated  for  their  importance  to 
job  success  as  in  the  previous  form.  This  general  format  for  collecting  such  data  has  the  advantage  of 
being  both  relatively  simple  and,  as  the  results  will  reveal,  sensitive  enough  to  describe  aspects  of  the 
management  positions  within  the  Corps  of  Engineers. 

Data  Collection 

Although  evaluating  the  same  number  of  individuals  in  each  of  the  GS  levels  would  have  been 
desirable,  difficulties  encountered  during  the  course  of  study  made  this  unfeasible.  A  sample  of  GS-1  Is 
through  GS-1 6s  was  originally  requested,  because  of  the  limited  number  of  GS-1 5s  and  -16s,  all  person¬ 
nel  at  these  high  management  levels  were  to  participate  in  the  study  to  provide  relatively  stable  esti¬ 
mates  of  the  managerial  job  at  these  levels.  The  participants  from  GS-11  through  GS-1 4  were  ran¬ 
domly  selected  from  the  Corps  Stratification  Tape  at  the  OCE  Manpower  Office.  All  Corps  GS-1 5s  and 
-16s  in  the  United  States  were  asked  to  participate.  All  participants  were  to  return  data  anonymously  by 
mail.  After  receiving  JAD  forms,  participants  were  sent  only  one  follow-up  request  for  completion  and 
return  of  the  forms. 

The  final  sample  selected  and  analyzed  during  the  study  is  described  in  Table  2.  As  shown  in  this 
table,  the  percentage  return  and  absolute  sample  size  is  somewhat  low  for  two  of  the  GS  levels  --  1 1 
and  16.  Better  return  rates  were  obtained  lor  the  levels  where  the  individual  enters  into  and  is  pro¬ 
moted  up  the  managerial  ladder,  i.e.,  GS-1 2  through  GS-1 5.  Since  the  forms  came  from  essentially  all 
GS-1 5s  and  -16s  within  the  Corps,  the  data  should  be  generally  representative  of  managerial  positions 

Table  2 

Description  of  the  JAD  Sample 


Job  Level 

Number 

Distributed 

Number 

Processed 

Percent 

Returned1 

GS  11 

80 

44 

59 

GS  12 

84 

65 

81 

GS  13 

79 

54 

80 

GS  14 

70 

49 

79 

GS  15 

272 

162 

65 

GS-16 

58 

20 

38 

Total 

643 

394 

68 

*This  column  includes  data  received  too  late  to  be  processed 


'  J  K  Hemphill,  ed.,  The  Engineering  Study  (Educational  Testing  Service,  1963);  J.  K.  Hemphill,  "Job  Descriptions  for  Execu¬ 
tives,"  Harvard  Business  Review,  Vol  37  (1959),  pp  5S-67. 
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within  the  Corps,  although  some  caution  is  justified  in  interpreting  the  results  because  of  the  moderate 
return  rate. 

Data  Analysis 

The  data  analysis  procedures  were  relatively  simple.  When  the  data  were  received,  all  information 
was  keypunched  and  mean  scores  for  time  spent  were  computed  for  each  GS  level.  These  mean  scores 
were  then  compared  to  determine  what  differences,  if  any,  were  characteristic  of  the  different  GS  level 
positions.  These  data  were  analyzed  with  an  analysis  of  variance  (ANOVA)  program  to  determine  the 
significant  differences  between  the  mean  percent  of  time  reported  by  the  different  managerial  levels. 

A  similar  procedure  was  followed  for  the  importance  ratings  of  the  various  job  activities.  The 
mean  importance  ratings  were  computed  for  each  GS  level,  and  these  were  compared  for  significant 
differences  by  the  ANOVA  program.  In  addition,  some  comparisons  of  certain  job  titles  and  of  current 
job  area  te  g.,  civil  works,  military  construction,  engineering,  construction)  were  made. 


Results 

The  Managerial  Job  A  na  'vsis  of  Time  Spent 

Instructions  for  the  JAD  advise  the  participant  that  not  all  aspects  of  managerial  jobs  are  included 
on  the  form  and  that  the  various  job  activities  are  not  necessarily  independent  (e  g..  Interpersonal 
Interaction  overlaps  with  Train  and  Motivate  personnel).  Nevertheless,  the  different  categories  of  job 
activities  do  provide  important  information  about  how  Corps  personnel  spend  their  time.  The  results  of 
this  part  of  the  study  are  presented  in  Table  3,  which  shows  the  average  percent  of  time  spent  by  each 
of  the  GS  levels  on  each  of  the  major  job  categories  included  in  the  JAD.  For  example,  all  levels  of 
managers  responded  similarly  to  Category  1,  Review  and  Evaluate  Projects.  The  average  time  spent  on 
these  activities  varied  from  14  to  18  percent,  and  there  were  no  significant  differences  among  the 
managerial  levels  as  determined  by  the  ANOVA  test.  In  this  case,  "statistical  significance"  shown  as 
p  <  05  means  that  identical  results  would  be  obtained  95  times  out  of  100,  p  <  01  is  99  times  out  of 
100 

Somewhat  similar  results  were  obtained  for  Category  2  --  Allocate  Resources.  Managers  devoted 
less  time  to  activities  in  this  category;  the  average  time  estimates  ranged  from  7  to  12  percent 

Different  levels  of  management  did  describe  their  jobs  differently  under  the  thiid  category  of  job 
activities  --  Train  and  Motivate  Personnel  The  mean  percent  of  time  spent  on  activities  in  this 
category  varied  from  4  percent  for  GS- 1 1  personnel  to  12  percent  for  GS-16s.  These  results  indicate 
the  gradually  increasing  scope  of  this  activity  as  one  proceeds  up  the  management  ladder. 

Differences  were  also  evident  in  the  amount  of  time  different  levels  of  management  spent  on  the 
next  category  -  Initiating  Improvements  in  Methods  and  Operations.  Although  the  differences  were 
not  marked,  ranging  from  only  7  to  1 1  percent,  they  were  statistically  significant. 

The  category  which  took  up  the  most  time  of  Corps  personnel  was  the  Use  of  Skills  and 
Knowledge  in  a  Technical  Capacity.  The  average  GS-11  reported  that  he/she  spent  approximately  38 
percent  of  his/her  time  on  such  job  activities.  By  the  time  managers  had  reached  GS-16,  only  13  per¬ 
cent  of  their  time  was  spent  on  these  activities.  These  results  indicate  that  less  time  is  spent  using 
technical  skills  as  an  individual  is  promoted  up  the  managerial  ladder. 

Category  6,  Interpersonal  Interaction,  showed  that  there  was  a  tendency  for  higher  level 
managerial  personnel  to  spend  more  time  in  interpersonal  interactions. 

Category  7,  Interactions  With  Other  Organizations,  did  not  differentiate  among  the  various 
managerial  levels  on  amount  of  time  spent.  The  average  amount  of  time  spent  interacting  with  person¬ 
nel  from  other  organizations  varied  from  12  to  17  percent. 

Category  8,  Scheduling  Work  of  Organizational  Components,  required  a  relatively  small  amount 
of  time  (ranging  from  6  to  1 1  percent)  across  all  managerial  levels. 

Category  9,  Long-Range  Planning  and  Control,  took  up  very  little  time  of  the  GS-1  Is  (3  percent), 
but  an  increasing  amount  of  time  as  one  progressed  up  the  managerial  ladder.  At  the  GS-16  level. 


Table  3 


Percent  of  Time  Spent  on  Different  Job  Activities 
Across  GS  Levels 


Job  Activity  Category 

Significant 

Average  Percent  of  Time 
Spent  at  Each  GS  Level 

11  12  13  14  IS 

1  Review  and  Evaluate 

Projects 

No 

14 

14 

16 

18 

14 

2  Allocate  Resources 

No 

7 

9 

11 

9 

10 

3.  Train  and  Motivate 

Personnel 

Yes** 

4 

7 

10 

9 

10 

4.  Initiate  Improvements  in 
Methods  and  Operations 

Yes** 

7 

7 

7 

7 

11 

5.  Use  Knowledge  and  Skills 
in  a  Technical  Capacity 

Yes** 

38 

41 

28 

20 

21 

6  Interpersonal  Interaction 

Yes* 

10 

10 

9 

12 

14 

7.  Interact  with  Other 

Organizations 

No 

15 

13 

14 

12 

14 

8.  Schedule  Work  of 

Organizational 

Components 

Yes* 

6 

11 

12 

12 

10 

9  Long-Range  Planning  and 
Control 

Yes" 

3 

5 

6 

8 

10 

•  p  <  .05 

•*p  <  01 

approximately  13  percent  of  the  time  spent  was  on  these  activities.  The  results  were  highly  significant 
in  differentiating  among  managerial  levels. 

Charting  the  response  patterns  of  three  categories,  under  which  the  other  six  can  be  grouped.  Fig¬ 
ure  4  illustrates  the  relationships  between  GS  level  and  percent  of  time  spent  on  the  JAD  categories, 
in  this  figure,  one  category  (Use  Knowledge  and  Skills  in  a  Technical  Capacity)  begins  at  a  high  level 
for  the  lower  GS  grades  and  decreases  as  the  GS  grade  increases.  This  category  deals  with  the  skills 
that  the  engineer  would  learn  in  college  and  for  which  the  engineer  was  first  employed  by  the  Corps. 
These  skills,  however,  are  used  less  as  the  engineer  advances  into  management  positions. 

A  second  relationship  is  exemplified  by  the  Review  and  Evaluate  Projects  category.  Here,  the 
amount  of  time  spent  is  fairly  constant  across  all  GS  levels.  These,  then,  are  skills  that  are  needed 
when  the  engineer  is  hired  and  remain  necessary  as  the  employee  progresses  up  the  managerial  ladder. 
Other  JAD  categories  belonging  to  this  group  include  Initiate  Improvements  in  Methods  and  Opera¬ 
tions.  Interact  with  Other  Organizations,  and  Schedule  Work  of  Organizational  Components. 

The  third  relationship  is  exemplified  by  the  Long-Range  Planning  and  Control  category.  Other 
JAD  categories  belonging  to  this  group  include:  Allocate  Resources,  Train  and  Motivate  Personnel, 
and  Interpersonal  Interaction.  The  amount  of  time  spent  on  these  categories  increases  as  GS  level 
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GS  LEVEL 

Figure  4.  Percent  of  time  spent  on  three  example 
categories  of  activities  by  GS  level. 

increases.  The  categories  in  this  group,  then,  could  be  the  subject  of  training  for  the  Corps  employee, 
because  the  typical  engineer  would  not  be  expected  to  have  developed  many  of  the  skills  necessary  to 
perform  these  functions  effectively.  Experience  and  in-service  training  would  be  necessary  to  prepare 
the  engineer  to  perform  competently  in  these  areas.  Since  the  future  of  the  Corps  is  at  least  in  part 
dependent  on  how  well  these  activities  are  carried  out,  it  appears  crucial  that  personnel  systematically 
develop  these  skills. 

Importance  of  Activities  by  GS  Level 

The  analyses  of  the  JAD  data  have  indicated  that  certain  activities  are  more  important  to  Corps 
personnel  at  higher  GS  levels.  The  importance  of  the  job  activities  to  the  GS  levels  is  summarized  in 
Table  4,  which  shows  those  activities  rated  very  important  by  th  1  personnel  completing  the  survey.  A 
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Table  4 


Job  Activities  Rated  as  Very  Important  by  Different  GS  Levels 

GS  Level 

11  12  13  14  15 

Review  and  Evaluate  Projects 

Monitor  Project  Progress  .  X  X  X  X 

Evaluate  Productivity  Measures  and  Systems . 

Review  Cost  Accounting  Records . 


Allocate  Resources 

Budget  Program  Expenditures .  X 

Allocate  Resources  to  Projects  and  Programs .  X  X  X 

Prepare  Cost  Estimates  for  a  Project . 


Train  and  Motivate  Personnel 

Analyze  Training  and  Developmental  Needs  of 


Subordinates . . 

Guide  Personnel  Assignments  and  Procedures .  X 

Provide  for  Personnel  Development  and 

Recognition .  X  X  X  X 


Initiate  Improvements  in  Methods  and  Operations 

Develop  and/or  Implement  New  Methods  of  Operation 
Plan  and  Initiate  New  Organizational  Procedures 
Design  New  Management  Procedures . 


Use  Knowledge  and  Skills  in  a  Technical  Capacity 

Solve  Various  Engineering  and  Scientific 

Problems .  X  X  X 

Prioritize  System  Requirements  and  Objectives .  X 

Act  as  Consultant  for  Solving  Technical  Problems  X 

Interpersonal  Interaction 

Help  Others  Resolve  Differences .  X  X 

Stimulate  Cooperation  Among  Employees .  X  X  X  X 

Provide  Feedback  to  Subordinates  About  Job 

Performance .  X  X 

Interact  With  Other  Organizations 

Negotiate  with  Outside  Organizations .  X  X 

Develop  Cooperative  Relationships  with  Clients, 

Interest  Groups,  etc .  X  X  X  X 

Keep  Up  With  Current  Events  Through  Contacts 

With  Outsiders . 


Schedule  Work  of  Organizational  Components 

Organize  Activities  of  Others 

to  Respond  to  Problems .  X 

Organize  Work  Schedules .  X 

Establish  Project  Priorities .  X  X  X  X 

long- Range  Planning  and  Control 

Participate  in  Long-Range  Planning  Meetings .  X 

Investigate  Possible  Future  Developments  and 

Response  Options .  X 

Develop  Long-Range  Priorities .  X 


Military  officers  were  omitted  from  the  sample;  however,  they  did 
occupy  a  majority  of  executive  positions. 
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"very  important"  activity  was  operationally  defined  as  one  rated  an  average  of  3.9  or  above  on  the  JAD 
importance  scale.  The  "very  important"  was  derived  by  combining  the  top  two  categories  ( 1  =  Unim¬ 
portant,  2  =Some  Importance,  3=Moderate  Importance,  4=Quite  Important,  5=Crucial.  Table  4 
allows  a  comparison  of  the  job  activities  which  were  common  across  the  GS  levels  and  summarizes  the 
increased  importance  of  different  activities  as  the  individual  progresses  up  the  management  ladder.  The 
table  also  illustrates  a  number  of  key  issues  with  which  this  chapter  has  dealt. 

For  GS-lls  or  -12s,  the  only  activity  rated  very  important  was  Solve  Various  Engineering  and 
Scientific  Problems.  If  the  jobs  of  the  individuals  holding  these  positions  are  restricted  to  engineering 
activities,  then  these  individuals  will  not  have  an  opportunity  to  develop  skills  that  are  so  valuable  for 
higher  level  positions.  Equally  or  more  important,  without  having  an  opportunity  to  develop  those 
skills,  engineers  also  will  not  have  an  oppo.  tunity  to  display  them;  and  those  skills  are  the  very  factors 
which  should  be  the  basis  for  selection  to  GS-13  positions. 

Table  4  clearly  shows  that  at  the  GS-13  level  a  number  of  new  managerial  duties  are  introduced, 
having  been  rated  very  important  by  the  incumbents  of  that  level.  At  higher  GS  levels,  the  job  activi¬ 
ties  of  importance  tend  to  move  away  from  strictly  engineering-oriented  activities  and  toward 
managerial  functions.  For  example,  activities  in  both  Interpersonal  Interaction  and  Train  and  Motivate 
Personnel  were  rated  as  very  important  for  the  first  time  at  the  GS-13  level.  If  GS-13  level  personnel 
and  higher  have  not  had  an  opportunity  to  develop  their  talents  in  these  new  duties,  then  their  perfor¬ 
mance  will  be  less  effective  than  it  could  or  should  be. 

The  table  also  shows  that  the  selection  of  people  for  higher  level  positions  should  not  be  based 
only  on  previous  job  performance,  but  rather  should  be  based  on  the  individual’s  performance  of  activi¬ 
ties  that  are  important  in  a  managerial  job.  And  the  JAD  form,  or  a  variation  of  it,  can  serve  as  an 
instrument  to  help  define  the  managerial  activities  of  an  open  position. 

By  the  time  individuals  advance  to  GS-14,  some  additional  activities  of  importance  are  typically 
added.  These  include:  Budget  Program  Expenditures;  Allocate  Resources  to  Projects  and  Programs. 
Help  Others  Resolve  Differences;  Provide  Feedback  to  Subordinates  About  Job  Performance;  Negotiate 
With  Outside  Organizations;  and  Organize  Activities  of  Others  to  Respond  to  Problems.  At  GS-14, 
Solve  Various  Engineering  and  Scientific  Problems  was  not  rated  as  very  important.  The  real  transition 
from  an  engineer  to  a  manager  has  clearly  begun  by  the  time  the  individual  reaches  the  GS-14  level. 

By  the  time  individuals  reach  the  GS-16  position,  important  job  activities  are  even  more  varied. 
The  activities  concerned  with  Long-Range  Planning  and  Control  in  particular  were  rated  very  important 
by  the  GS-16s.  even  though  they  had  not  been  so  rated  by  lower  levels.  Thus,  sensitivity,  awareness, 
and  competence  in  these  activities  should  be  significant  in  the  selection  of  GS-16s. 

Only  one  category  of  activity  was  not  rated  very  important  by  any  GS  level:  Initiate  Improve¬ 
ments  and  Methods  in  Operations.  Since  the  activities  in  this  category  were  concerned  with  organiza¬ 
tional  change,  it  seemed  unusual  that  such  activities  were  not  viewed  as  more  important  -  particularly 
if  it  is  assumed  that  continuing  change  will  characterize  future  development  of  the  Corps. 

Five  activities  were  rated  very  important  by  GS-13s  through  GS-16s  -  in  other  words,  important 
regardless  of  the  level  of  the  managerial  job. 

1  Monitor  Project  Progress 

2.  Provide  for  Personal  Development  and  Recognition 

3.  Stimulate  Cooperation  Among  Employees 

4.  Develop  Cooperative  Relationships  With  Clients,  Interest  Groups,  etc. 

5.  Establish  Project  Priorities. 

Table  4  does  not  necessarily  highlight  job  activities  that  change  rapidly  in  importance  as  one 
proceeds  up  through  management  (i.e.,  those  activities  that  have  the  highest  relationships  to  grade 
level).  Although  there  were  a  number  of  specific  activities  with  marked  differences  across  grade  level, 
those  most  likely  to  change  in  importance  included: 

1 .  Allocate  Resources  to  Projects  and  Programs 

2.  Provide  for  Personnel  Development  and  Recognition 
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3.  Develop  Long-Range  Priorities 

4.  Investigate  Possible  Future  Developments  and  Response  Options 

5  Provide  Feedback  to  Subordinates  About  Their  Job  Performance. 

Job  Assignment  Analyses 

On  the  front  of  the  JAD  form,  each  respondent  was  asked  to  list  his/her  current  job  title  (for 
example,  civil  engineer,  electrical  engineer,  structural  engineer,  economist),  his/her  divisional  assign¬ 
ment  (engineering,  planning,  construction,  etc.),  and  whether  he/she  worked  in  the  area  of  Civil 
Works  or  Military  Construction.  The  respondents  were  grouped  into  these  various  categories  and  their 
responses  to  the  JAD  analyzed  to  determine  if  there  were  significant  differences  in  the  amount  of  time 
spent  on  and  the  importance  of  the  job  characteristics. 

These  analyses  were  performed  because  interview  results  with  Corps  personnel  indicated  contro¬ 
versies  about  prerequisites,  advantages,  and  limitations  of  various  cross-training  assignments.  The 
results  of  the  current  study  revealed  that  on  the  demographic  measures  examined,  the  job  title,  the 
divisional  assignment,  and  the  respondent’s  involvement  in  Civil  Works  or  Military  Construction  were 
not  differentiating  factors. 

Positions  at  similar  levels  within  the  Corps  do  have  many  managerial  activities  that  are  similar  in 
importance  —  at  least  with  respect  to  the  amount  of  time  spent  on  them.  Positions  at  different 
managerial  levels  (GS-11  vs  GS-15  for  example),  however,  are  different  both  in  the  amount  of  time 
spent  on  specific  activities  and  in  the  importance  of  those  activities. 

Job  Activity  Categories  for  GS-15s  and  -  lbs 

This  section  analyzes  the  total  group  of  GS-15s  (N  =  162)  and  -16s  (N  =  20)  who  responded  to 
the  JAD.  and  identifies  skills  necessary  for  performance  at  those  levels. 

In  Table  5,  the  JAD  categories  are  rank-ordered  according  to  the  average  percent  of  time  spent  on 
each  category  by  the  GS-15s  and  -16s.  In  the  second  column,  the  average  rate  of  importance  of  the 
JAD  categories  is  presented.  The  numbers  in  parentheses  represent  the  rank-order  of  the  average 
importance. 

Category  5,  Use  Knowledge  and  Skills  in  a  Technical  Capacity,  was  the  unquestionable  leader  in 
percent  of  time  spent  for  the  GS15s  and  16s:  20.1  percent.  The  next  three  categories  are  grouped 
rather  closely  14.1  to  14.3  percent.  Then  there  is  another  sizable  drop;  the  last  five  categories  are 
within  the  range  of  one  percentage  point. 

The  Average  Rated  Importance  column  of  Table  5  also  presents  some  interesting  findings.  The 
relationship  between  average  rated  importance  and  percent  of  time  spent  was  not  high,  which  indicated 
that  the  priorities  of  the  GS-I5s  and  -16s  were  not  well  reflected  in  the  amount  of  effort  expended. 
That  is,  the  category  with  the  highest  average  rated  importance  —  Interpersonal  Interaction  -  did  not 
have  the  most  time  devoted  to  it;  of  course,  it  is  also  one  in  which  professional  engineers  typically  have 
received  little  formal  training.  This  category  has  been  identified  before  as  one  in  which  opportunities 
for  individual  development  should  be  provided,  and  one  which  should  be  carefully  considered  in  the 
selection/promotion  process. 

The  second  and  third  most  important  categories  -  Interact  With  Other  Organizations,  Train  and 
Motivate  Personnel  -  are  also  definitely  managerial  in  nature.  These  findings  reinforce  the  idea  that 
the  best  engineers  may  not  be  the  most  qualified  for  the  higher  GS  level  position. 

These  results  from  the  JAD  highlight  the  importance  of  carefully  analyzing  each  job  opening 
before  the  selection  process.  After  this  analysis,  the  individuals  who  can  handle  best  the  various  activi¬ 
ties  of  higher  level  positions  should  be  selected.  The  person  who  has  not  developed  managerial  skills 
should  not  be  encouraged  to  apply  for  higher  managerial  positions.  Other  methods  should  be  con¬ 
sidered  for  recognizing  the  talents  of  those  who  remain  engineers  --  for  example,  placing  such  individu¬ 
als  in  divisional  positions  which  may  be  more  in  line  with  their  capabilities,  rather  than  in  managerial 
spots  which  may  not  match  their  abilities  and  interests.  Selecting  the  wrong  person  for  a  managerial 
position  could  be  detrimental  to  the  individual,  subordinate  employees,  and  the  Corps. 
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Table  5 


Analysis  of  Job  Activity  Categories  by  Time  Spent,  and  Average  Rated 
Importance  for  GS  Levels  15  and  16 


Percent 

Average  Rated 

Category 

of  Time 

Importance* 

Use  Knowledge  and  Skills  in 
a  Technical  Capacity 

20.1 

3  57  (6) 

Interact  with  Other  Organiza¬ 
tions 

14.3 

3.85  (2) 

Interpersonal  Interaction 

14  I 

4.00  (1) 

Review  and  Evaluate  Projects 

14  1 

3.34  (8) 

Initiate  Improvements  in  Methods 
and  Operations 

109 

3.52  (7) 

l  ong  Range  Planning  and  Control 

103 

3  80  (4) 

Train  and  Motivate  Personnel 

10.2 

3.83  (3) 

Allocate  Resources 

10.2 

3.30  (9) 

Schedule  Work  of  Organizational 

Components 

10.1 

3.73  (5) 

'Numbers  in  parentheses  represent  rank  order  for  rated  importance 

The  following  basic  issues  are  evident  from  data  generated  by  the  JAD. 

1 .  Personnel  at  the  higher  managerial  levels  within  the  Corps  generally  cannot  be  specialists  and 
function  well.  They  must  have  abilities  across  all  categories  investigated.  If  a  person  wishes  to  spend 
his/her  time  primarily  in  a  given  specialty,  then  that  person  usually  should  not  seek  a  managerial  posi¬ 
tion. 

2.  The  evidence  suggests  that  even  the  GS-15s  and  16s  may  think  of  themselves  as  engineers 
first  and  managers  second.  This  is  reflected  in  the  high  percent  of  time  spent  on  category  5.  The  wis¬ 
dom  of  the  15s  and  16s  spending  this  disproportionate  amount  of  time  on  a  category  ranked  sixth  in 
importance  appears  questionable. 


Summary  and  Conclusions 

The  Job  Activities  Description,  a  brief  form  to  collect  information  about  the  importance  of  and 
time  spent  on  various  job  duties,  was  administered  to  394  U.S.  Army  Corps  of  Engineers  employees  at 
GS  levels  1 1  through  16.  The  JAD  focused  on  management  practices  that  were  generally  characteristic 
of  key  positions  within  the  Corps.  The  findings  were: 

1.  The  study  clearly  demonstrated  the  feasibility  of  using  descriptions  of  job  activities  to  help 
guide  selection  decisions.  The  importance  of  each  of  27  activities  was  presented  for  the  grade  levels 
studied.  At  grades  11  and  12,  no  management  activities  were  rated  very  important  other  than  Use 
Knowledge  and  Skills  in  a  Technical  Capacity.  At  grade  13,  eight  activities  were  rated  very  important, 
at  grade  16,  ten  activities. 

2.  The  amount  of  time  spent  on  various  job  activities  also  varied  by  GS  level.  In  general,  higher 
graded  personnel  performed  manager-generalist  roles,  spending  their  time  on  many  different  activities 
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Individuals  at  lower  grade  levels  spent  most  of  their  time  in  a  specialist  role  performing  technical  duties 
related  to  their  specialty. 

3.  In  addition  to  the  general  findings  about  the  importance  of,  and  time  spent  on  various  activi¬ 
ties  across  grade  levels,  the  activities  description  approach  should  be  used  to  further  define  the  most 
important  specific  job  activities  for  each  position  opening. 

4.  Selection  should  not  be  based  only  on  past  performance,  but  should  particularly  emphasize  the 
potential  of  candidates  in  performing  those  activities  that  are  important  in  the  open  position. 

5.  Individuals  at  all  grade  levels  should  be  given  opportunities  to  develop  and  display  their 
management  potential.  Such  opportunities  should  then  facilitate  more  effective  selection  decisions. 


4  RANKING  AND  RATING  PROCEDURES 


Before  the  selection  and  promotion  process  for  Corps  managers  can  be  improved,  a  better  system 
for  assessing  the  proficiency  (i.e.,  referral  qualifications)  of  current  personnel  is  needed  The  Corps 
now  uses  the  SKAP  system,  but  since  there  are  problems  with  it  (Chapter  2),  this  study  has  developed 
other  procedures  which  are  proposed  for  rating  and  ranking  employees. 


Performance  Evaluation  Rationale 

Performance  evaluation  systems  have  several  purposes,  which  fall  into  two  broad  categories:  to 
assess  employees’  development  needs  and  to  aid  management  decisions  regarding  personnel  In  the 
first  category,  evaluation  activities  are  used  to  identify  specific  training  requirements  of  individual 
employees,  act  as  stimuli  to  insure  that  supervisors  observe  and  coach  their  subordinates  more  closely, 
and  help  motivate  employees  by  providing  accurate  feedback  concerning  work  performance. 

In  the  second  category,  evaluation  activities  are  an  indispensable  part  of  any  decision-making  pro¬ 
cess  regarding  promotions,  layoffs,  and  organizational  rewards.  Evaluation  activities  are  also  important 
as  a  reference  base  for  research  into  various  management  actions. 

As  many  as  90  percent  of  all  companies  have  some  kind  of  employee  evaluation  system.'’  There 
is  a  strong  trend  toward  results-oriented  approaches  -  such  as  Management  by  Objectives  --  often  in 
combination  with  the  more  traditional  global  or  trait  ratings  of  employees.  The  evaluation  system  can 
accommodate  peer  ratings,  supervisor  ratings,  and  committee  reports  at  higher  levels.  Companies  with 
manager  replacement  tables  (lists  of  potential  candidates)  tend  to  have  committees  or  panels  handling 
the  selection  process.  These  committees  may  consider  performance  evaluation  information,  but 
apparently  never  as  a  sole  criterion.  Some  of  the  larger  companies  use  two  or  more  different  evaluation 
systems  for  different  purposes.  For  example,  one  system  may  be  designed  to  allocate  organizational 
rewards  1  Another  might  provide  opportunities  for  personal  or  professional  development. 

Evaluations  focusing  on  each  of  the  two  purposes  -  assessing  development  needs  and  aiding  per¬ 
sonnel  decisions  -  could  be  done  separately,  but  the  information  provided  by  one  should  be  available 
in  any  managerial  decision  concerning  the  other.  Because  of  the  evaluation’s  potential  impact  on  an 
employee,  feedback  to  the  person  being  rated  should  be  an  important  element  of  any  performance 
evaluation  system,  yet  has  received  only  minimal  attention.  Furthermore,  employees  should  receive 
feedback  about  their  performance  at  any  appropriate  time,  not  just  during  formally  mandated  periods 

Employee  performance  can  be  measured  in  a  number  of  ways.  Some  of  the  more  common  tech¬ 
niques  include: 

1  Essay  appraisals,  requiring  the  rater  to  describe  the  individual’s  strengths,  weaknesses,  and 
potential.  These  essay  appraisals  are  usually  not  standardized  and  are  very  difficult  to  compare 

2  Rating  scales,  requiring  the  rater  to  assess  the  individual’s  performance  and  traits  on  a  number 
of  standardized  scales. 

3.  Field  reviews,  requiring  the  rater  to  meet  with  other  evaluators  and  a  personnel-staff  member 
to  reach  a  consensus  about  the  individual’s  ratings. 

4.  Forced-choice  ratings,  requiring  the  rater  to  choose  sets  of  standardized  statements  that  are 
most  descriptive  or  least  descriptive  of  the  individual. 

5.  Rankings,  requiring  the  rater  to  rank-order  several  individuals  along  a  standardized  dimension 
of  performance. 

6.  Critical  incidents,  requiring  the  rater  to  keep  a  diary  of  specific  instances  of  the  individual  s 
performance. 

7  Work  standards,  requiring  the  rater  to  compare  the  individual’s  achievements  with  predeter¬ 
mined  standards  of  performance.  This  is  very  much  a  results-oriented  approach. 

h  I  B  Miner  and  M  Ci  Miner,  Personnel  and  Industrial  Relations:  A  Managerial  Approach  (MatMillun.  1*177). 

J  P  Campbell.  M  I)  Dunnette,  F  I  Lawler  III,  and  K  F.  Wen.lt.  Jr  ,  Managerial  Behavior.  Performance.  and  l.lh\ioene\y 
(McGraw-Hill.  1970),  pp  IIJ  II6 


8  Management  by  objectives,  requiring  (he  rater  (o  meet  with  the  individual  to  set  goals  and 
later  evaluate  whether  those  goals  were  achieved.  This  is  also  a  results-oriented  approach 

9  Assessment  centers,  usually  requiring  the  individual  to  be  sent  off  the  job  to  a  series  of  struc¬ 
tured  exercises  and  tests  which  evaluate  his/her  potential 

Each  of  these  techniques  has  different  advantages  and  disadvantages.  Table  6  shows  the  recom¬ 
mended  technique  for  a  given  purpose. 

There  is  no  one  "best"  performance  evaluation  system,  rather,  the  choice  of  which  is  best  depends  on 
the  system’s  intended  use.  However,  all  good  evaluation  systems  have  several  common  elements: 

1.  A  clear  statement  of  purpose.  The  purpose  of  the  evaluation  will  determine  what  techniques 
should  be  used  and  how  often. 

2.  A  job  analysis.  A  formal  job  analysis,  such  as  the  Job  Activities  Description,  covers  many  ele¬ 
ments  of  managerial  performance  -  elements  determining  activities  that  are  important  in  different  jobs. 
This  information  is  used  to  help  develop  standards  against  which  performance  can  be  compared.  Of 
course,  job  requirements  are  usually  multidimensional  and  involve  several  different  skills  and 
behaviors  Because  of  this,  and  the  need  for  the  system  to  be  accepted  and  considered  fair,  employees 

Table  6 

Recommended  Evaluation  Techniques  for  Different  Purposes 

(Adapted  from  W  Oberg,  "Making  Performance  Appraisal  Relevant," 

Harvard  Business  Review,  Vol  SO,  No.  1  [1972],  pp  61-67.) 


TECHNIQUE 

Forced-  Management 

Rating  Choice  Field  Critical  Work-  By  Assessment 

Purpose:  Scales  Rankings  Ratings  Reviews  Incidents  Standards  Objectives  Centers 


Provide  base  for 
management  decisions 
rewards,  promotions. 

etc  X  \  X  X  \ 


Identity  potential, 
identify  training  and 

development  needs  X  X 


Prod  supervisors  to 
coach  and  observe 
more  closely 


Motivate  workers 
through  feedback 


Reference  base  lot 
personnel  tesearch 
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incumbent  in  the  position  should  be  encouraged  to  offer  suggestions  as  performance  standards  are 
developed. 

3.  Factors  to  be  evaluated:  outcomes,  traits,  or  behaviors  Outcomes  are  the  direct,  ultimate 
concern  of  the  organization:  witness  the  increase  in  results-oriented  evaluation.  However,  most  out¬ 
comes  depend  on  many  factors,  only  some  of  which  can  be  controlled  by  the  employee  being  evaluated 
Outcomes  may  also  take  a  long  time  to  become  apparent. 

Traits  such  as  "enthusiasm"  or  "ability  to  communicate"  are  more  immediate  and  are  presumable 
more  under  the  control  of  the  individual  However,  they  are  often  only  vaguely  defined  and,  more¬ 
over,  are  almost  always  inferred  from  the  person’s  actual  behavior 

Actual  job  behaviors,  on  the  other  hand,  are  directly  observable,  clearly  definable,  and  presumably 
related  fairly  closely  to  measurable  outcomes.  Behavior-oriented  measures,  such  as  coworker  coopera¬ 
tion,  are  preferable  to  others  and  should  be  the  measurement  of  choice.  However,  useful  measures  of 
outcomes  or  traits  can  be  important  adjuncts  to  those  of  behavior  and  should  be  considered  for  the 
different  kinds  of  information  provided. 

4.  Designation  of  the  raters  or  evaluators  Careful  consideration  must  be  given  to  this  element  of 
the  evaluation  system.  Generally,  more  than  one  rater  should  be  available  for  each  employee  Indivi¬ 
duals  can  be  effectively  evaluated  by  their  peers  as  well  as  by  higher  levels  of  management  Having 
multiple  raters  requires  a  method  of  collating  the  information  about  each  employee. 

5.  Training  for  raters  or  evaluators.  Ratings  are  found  to  be  more  accurate  when  the  raters  have 
received  training.  Therefore,  training  has  an  impact  on  the  reliability  of  the  system  and  should  be  rou¬ 
tinely  built  into  any  performance  evaluation  system 

6.  Criteria  of  performance.  The  evaluation  system  must  measure  comprehensive  and  relevant 
criteria  of  performance  or  employees  will  not  accept  the  system.  The  validity  of  the  evaluation  system 
can  be  improved  by  the  job  analysis  and  by  the  input  from  the  employees  occupying  a  given  position 

7.  Extent  of  feedback.  Feedback  is  generally  desirable  because  it  can  help  motivate  employees; 
however,  providing  performance  feedback  can  require  a  high  degree  of  tael  and  interpersonal  skill 
Supervisors  typically  are  reluctant  to  confront  an  employee  and  make  negative  comments  about  an 
individual's  performance.  Nonetheless,  any  evaluative  system  should  require  that  employees  receive 
feedback  about  their  performance  Supervisors  should  be  trained  in  offering  feedback  so  that  they  can 
provide  positive  and  constructive  information  that  the  employee  can  use  in  improving  his/her  perfor 
mance. 

8.  Timing  of  evaluations.  Evaluations  can  be  time  consuming  if  done  too  frequently,  but  they 
can  also  be  ineffective  if  not  done  frequently  enough.  A  suggested  interval  between  evaluations  is  6  us 
12  months;  however,  if  employees  are  highly  skilled  and  specialized,  and  if  tasks  do  not  follow  standard 
cycles,  then  employees  themselves  can  request  evaluation  reviews  on  an  unscheduled  basis 


First-Level  Supervisory  Assessment 

The  criterion  rating  form  to  be  filled  out  by  first-level  supervisors  is  fairly  straightforward.  Appen¬ 
dix  A  is  an  example  of  this  document.  All  ratings  for  one  employee  are  to  be  done  on  a  single  8  1/2  x 
1 1  in.  sheet  of  paper.  The  first  side  of  this  form  asks  the  supervisor  to  rate  the  employee  on  each  of 
the  five  criterion  dimensions.  A  paragraph  defines  each  dimension,  as  shown  in  Appendix  A  The 
supervisor  can  rate  the  employee  on  each  dimension  by  placing  a  mark  in  the  appropriate  box  on  a 
five-point  scale  The  first  point  of  the  scale  indicates  that  the  employee  is  weak  in  the  performance 
characteristic  and  would  fall  in  the  lower  10  percent  of  all  employees.  The  second  box  indicates  that 
the  employee  is  marginally  competent  on  the  performance  characteristic  and  would  fall  in  the  next  25 
percent  (i.e  ,  the  1 1th  to  25th  percentile).  The  middle  choice  communicates  that  the  employee  is  fully 
competent,  this  point  on  the  scale  should  represent  the  middle  30  percent  of  all  employees  The  fourth 
choice  would  indicate  that  the  employee  is  exceptionally  competent,  falling  in  the  next  25  percent  or 
the  66th  through  90th  percentile.  The  fifth  choice  would  indicate  that  the  employee  is  outstanding  in 
this  performance  characteristic  --  one  of  the  top  10  percent  of  all  employees.  The  supervisor  is 
instructed  not  to  let  the  employee’s  ratings  on  one  dimension  influence  ratings  on  other  dimensions 
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The  second  side  of  the  Immediate  Supervisory  Assessment  Form  lists  the  24  SKAP  elements. 
The  supervisor  is  instructed  to  indicate  the  four  elements  on  which  the  employee  is  the  most  highly 
skilled  and  the  four  elements  on  which  the  employee  has  the  greatest  opportunity  for  improvement. 
Historically,  the  ratings  made  on  the  SKAP  form  have  had  little  variability;  for  the  most  part,  employ¬ 
ees  have  been  rated  fully  competent  The  procedure  outlined  above  would  force  discrimination  on  the 
part  of  the  supervisor,  improving  diagnostic  sensitivity  and  making  the  assessment  more  useful  to 
employees’  career  development. 


Second* Level  Supervisory  Assessment 

The  final  major  criterion  form  for  the  study  collects  from  second-level  supervisors  ratings  on 
three  of  the  five  performance  characteristics.  Results  from  pilot  use  of  this  form  at  the  Sacramento 
District  show  that  collecting  independent  assessments  of  performance  from  second-level  supervisors  is 
feasible  and  perhaps  necessary  The  current  SKAP  procedure  requires  that  second-level  supervisors 
only  approve  the  ratings  which  have  been  made  by  first-level  supervisors.  The  use  of  a  form  like  that 
discussed  below,  or  the  independent  ratings  of  subordinates  by  second-level  supervisors  on  the  SKAP 
form,  provides  a  broader  assessment  base  and  additional  reliability  and  validity  to  the  entire  rating  pro¬ 
cess. 

The  Second-Level  Supervisor's  Rating  Form  (Appendix  B)  is  somewhat  more  elaborate  than  that 
of  the  first-level  supervisor.  It  allows  for  the  ratings  of  up  to  21  employees  on  a  single  form.  With  this 
instrument,  the  supervisor  rates  all  employees  on  a  single  dimension  before  proceeding  to  ratings  on 
the  next  dimension.  This  should  decrease  the  likelihood  that  the  second-level  supervisor  will  rate  cer¬ 
tain  employees  high  on  all  three  dimensions.  Each  dimension  corresponds  to  one  page  in  the  form,  and 
a  paragraph  description  tells  the  supervisor  completing  the  form  what  the  accompanying  11 -point  scale 
is  to  measure  Five  points  on  the  scale  have  a  behavioraily  based,  one-sentence  description  of  what 
performance  at  that  level  should  indicate.  A  hypothetical  breakdown  of  how  a  sample  of  100  employ¬ 
ees  might  be  distributed  across  the  descriptive  statements  is  also  provided  to  Mp  guide  the  supervisor 
in  discriminating  among  different  levels  of  performance. 


Peer  Evaluation  Rationale 

Performance  evaluation  through  the  use  of  peer  ratings  was  started  by  the  U  S.  Army's  "Buddy 
Rating"  system.  Since  that  time,  several  arguments  have  favored  such  a  peer  evaluation  system. 

1.  Coworkers  are  likely  to  perceive  different  aspects  of  performance  than  are  supervisors 
Behavior  among  coworkers  is  likely  to  be  different  and  more  "real"  than  behavior  in  the  presence  of  a 
supervisor. 

2  Coworkers  are  in  a  better  position  to  perceive  more  behavior  because  they  usually  have  more 
contact  with  the  employee  than  does  the  supervisor. 

1  Peer  ratings  can  be  used  as  a  check  on  the  reliability  of  performance  ratings  completed  by  the 
supervisor 

There  is  strong  evidence  that  peer  ratings  are  valid.  Peer  ratings  in  a  training  situation  have 
predicted  later  promotion  and  performance  appraisal  results  for  middle-level  managers  with  correlations 
in  the  40s  *  In  contrast,  the  ratings  by  the  training  staff  were  much  weaker  predictors.  Peer  ratings 
have  had  validities  in  the  30s  and  40s  in  predicting  success  in  training,  promotion,  and  job  perfor¬ 
mance.'’  One  author  concluded  that  peer  evaluations  were  useful  in  predicting  later  global  ratings  of 
success  in  managerial  jobs  within  industry.10  In  a  study  cited  earlier  by  Campbell,  et  al.,  it  was  noted 
that  the  correlations  between  peer  and  supervisory  ratings  are  only  moderate,  with  peer  ratings  being 


"  A  I  Kraut,  'Prediction  of  Managerial  Success  by  Peer  and  Training-Siaff  Ratings,'  Journal  of  Applied  Psychology,  Vol  60  (WM, 
pp  14-19 

"*  R  F  Carlson.  'The  Current  Status  of  Judgmental  Techniques  in  Industry,”  in  W.  C  Byham  and  l).  Bobin,  cds  ,  Alternatives  to 
Paper  and  Pencil  resting  (University  of  Pittsburgh,  Graduate  School  of  Business,  1973) 

111 J  Wen/.  'Selecting  Supervisors  with  Peer  Ratings.'  Personnel  Psychology,  Vol  1 1  (1958),  pp  25-3b 
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more  reliable  predictors  as  time  passes  and  the  employee  moves  into  different  positions."  So  in  addi¬ 
tion  to  the  validity  of  the  peer  ratings,  supervisory  ratings  and  peer  ratings  appear  to  measure  slightly 
different  aspects  of  work  performance  Whenever  possible,  ratings  should  come  from  more  than  one 
source  —  the  supervisor  alone  or  the  coworker  alone  would  omit  valuable  information  A  multi-trait, 
multi-rater  matrix  has  been  suggested  to  compare  the  ratings  from  different  sources.1-’ 

Organizations  have  not  used  peer  ratings  extensively  for  two  principal  reasons. 

1.  There  is  considerable  awareness  that  coworkers  often  view  their  peers  as  either  friends  or 
rivals.  This  could  lead  to  systematic  bias  in  the  peer  ratings  since  an  individual  could  gi\e  high  ratings 
to  friends  and  low  ratings  to  rivals. 

2.  There  is  a  possibility  for  collusion  --  coworkers  could  make  deals  to  rate  each  other  highly 

Research  on  these  two  problems  indicates  that  their  impact  is  minimal  One  author  found  that 
the  reliability  of  peer  ratings  was  typically  high  tin  the  .KOs>  and  that  the  validities  were  unaffected  by 
friendship  patterns.  Through  experimentation,  he  concluded  that  the  ratings  received  by  an  individual 
in  one  group  would  be  about  the  same  as  the  ones  he/she  would  receive  in  another  group."  Other 
authors  have  also  stated  that  the  fears  about  these  two  problems  are  greatly  exaggerated  and  that 
extraneous  factors,  such  as  appearance  or  friendship,  will  not  invalidate  the  findings  These  researchers 
decided  that  peer  ratings  can  be  used  effectively  even  when  the  raters  are  aware  that  their  ratings  will 
have  an  impact  on  the  ratee's  career.14 

Overall,  the  conclusion  is  that  peer  evaluations  would  serve  a  useful  purpose  and  would  be  a 
worthwhile  addition  to  the  ratings  by  supervisors.  The  effectiveness  of  the  peer  evaluations  could  be 
partially  dependent  on  personnel  procedures  allowing  (or  discouraging)  job  mobility  across  the  organiza¬ 
tion  The  more  people  in  an  organization  who  are  aware  of  an  individual’s  abilities,  the  more  likely 
that  reliable  and  valid  measurements  of  work  performance  by  coworkers  can  be  obtained 


The  Peer  Ranking  Process 

In  this  procedure,  described  in  Appendix  C,  all  engineers  and  scientists  at  CiS-12  and  above  would 
rank  on  two  criterion  dimensions  -  Technical  Competence  and  Communication  Ability  --  scientific  and 
engineering  personnel  at  CiS- 1 1  and  above  whom  they  know  well  enough  to  evaluate  All  scientific  and 
engineering  personnel  at  GS-12  and  above  would  be  invited  to  attend  a  meeting  During  the  meeting, 
each  person  would  choose  a  deck  of  randomly  selected  computer  cards,  each  with  the  name  of  one  indi¬ 
vidual  at  CiS- 1 1  or  above 

The  decision  to  use  only  the  Technical  Competence  and  Communication  Ability  aileron  dimen 
sions  was  based  on  several  coasiderations:  (1)  of  the  five  dimensions,  these  two  are  the  most  readily 
apparent  to  coworkers.  (2)  the  ranking  process  is  time-consuming,  (3)  more  than  two  numbers  on  the 
ranking  cards  would  be  difficult  to  analyze  given  the  possibility  of  the  rankers’  misplacing  numbers  on 
the  cards.  (4)  given  the  historically  high  reliability  of  peer  rankings,  two  were  seen  as  sufficient,  and  ( 5 ) 
the  supervisory  ratings  are  used  as  reliability  checks  on  the  rankings  by  peers 

Figure  5  illustrates  the  sequence  of  events:  the  first  step  of  the  ranking  process  is  for  each  person 
to  take  the  deck  he/she  has  received  and  sort  it  into  two  groups  --  those  whose  performance  is  known 
well  to  him/her  and  those  whose  performance  is  not.  Cards  with  names  of  subordinates  and  the  card 
with  the  individual’s  own  name  on  it  are  put  with  those  not  well  known  and  discarded  The  person 
then  ranks  those  whose  performance  is  well  known  from  high  to  low  based  on  his/her  interpretation  of 
their  technical  competence  After  ranking  the  cards  in  order,  the  evaluator  records  on  each  the  number 
corresponding  to  its  position  in  the  deck.  The  cards  are  then  shuffled,  and  the  process  of  ranking  the 

"  J  P  Campbell,  M  0  Dunneltc.  F  F  Lawler  III,  and  K  K  Weick.  Jr..  Managerial  Behavior.  Performance,  ami  t'/fkvnwww 
(McGraw-Hill.  1470I.  PP  1 1 3- 1 1 6 

1  -  F.  F  lawler.  III.* Management  Performance  as  Seen  from  Above.  Below,  and  Within?  Journal  of  Applied  Psychology.  Vol  M 
(|%7).  pp  247-25.1.  R  J  Klimoski  and  M  London.  "The  Role  of  the  Rater  in  Performance  Appraisal."  Journal  ol  ipphe.l 
Pmhofogv.'  Vol  59  1 1174).  pp  445-451 

"  F  P  Hollander,  "V  alidity  of  Peer  Nominations  in  Predicting  a  Distinct  Performance  Criterion."  Journal  of  Applnil  Ps»hi>fogi. 
Vol  49  <l%5>.  pp  4.14  458 

14  J  P  Campbell.  M  D  Dunncttc.  I  F  l.awlcr  til,  and  K  F  Weick,  Jr  .  Managerial  Rehavnir  Prrformamv  and  flfo  nwrnrw 
(McGraw-Hill.  I >470> 


35 


cards  and  recording  the  position  in  the  deck  is  repeated  for  an  assessment  of  the  employees’  perfor¬ 
mance  in  communications.  After  this  is  completed,  the  decks  are  returned  to  the  investigators  for  pro¬ 
cessing. 


Figure  5.  Ranking  process. 
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Once  all  the  data  have  been  collected,  they  are  organized  for  computer  processing,  which  involves 
a  variety  of  sophisticated  techniques  to  distill  from  the  full  data  matrix  a  single  rating  for  each 
employee  on  each  of  the  two  criterion  dimensions.  This  computerized  technique  obtains  an  average 
ranking  per  individual  based  on  the  data  matrix,  which  has  been  adjusted  to  reduce  the  effects  of  atypi¬ 
cal  rankers  and  remove  extreme  rankings  for  each  individual  ranked.  Another  outcome  of  the  compu¬ 
terized  technique  is  an  estimate  of  the  reliability  for  the  adjusted  rankings  of  each  dimension. 

This  process  results  in  an  acceptable  number  of  rankings  for  each  employee  at  GS- 1 1  or  above 
Rankings  by  several  evaluators  -  rather  than  one  or  two  -  produce  data  which  are  more  reliable  and 
therefore  produce  comparable  results  over  time. 


Criterion  Data 

The  specific  aspects  of  evaluation  by  immediate  supervisor,  second-level  supervisor,  and  peers 
have  been  detailed  on  the  previous  pages.  In  an  attempt  to  broaden  the  base  for  assessing  perfor¬ 
mance,  these  three  separate  sources  of  information  are  used  in  the  overall  criterion  package.  The  three 
sources  provide  rankings  or  ratings  of  an  employee's  performance  on  five  dimensions  (Table  7): 
Technical  Competence,  Communication,  Decision  Making,  Innovation,  and  Coordination.  Peers  are 
required  to  rank  only  persons  they  know  well  enough  to  evaluate,  while  each  supervisory  form  requests 
ratings  on  all  employees. 

When  data  collection  is  completed,  all  data  provided  by  the  various  forms  are  keypunched  onto 
computer  cards  to  put  the  information  in  machine-readable  form.  The  data  are  then  organized  into 
magnetic  tape  records  at  one  record  per  individual  employee.  Each  record  contains  the  data  based  on 
peer  rankings  for  two  job  dimensions,  on  immediate  supervisor’s  ratings  across  all  five  performance 
dimensions,  and  on  the  second-level  supervisory  ratings  for  three  performance  dimensions. 

To  cross-validate  portions  of  this  package  of  evaluation  criteria,  three  of  the  five  performance 
characteristics  discussed  here  are  also  content  categories  of  the  Biographical  Inventory  presented  in  the 
next  chapter.  Technical  Competence,  Communication,  and  Innovation. 


Summary 

Ratings  by  an  employee’s  immediate  supervisor  are  such  a  direct  measure  of  job  performance  that 
they  are  unquestionably  accepted  as  relevant  and  meaningful  in  describing  the  individual’s  job  perfor¬ 
mance  over  a  given  time  period  The  supervisor  is  generally  in  the  best  position  to  observe  and  report 

Table  7 

Dimensions,  Sources,  and  Methods  of  Criterion  Data 


Criterion  Dimensions: 

Peers 

Source 

Immediate 

Supervisor 

Second-Level 

Supervisor 

Technical  Competence 

Ranking 

Rating 

Communication 

Ranking 

Rating 

Rating 

Decision  Making 

Rating 

Rating 

Innovation 

Rating 

Rating 

Coordination 

Rating 

on  employee  behavior;  yet  he/she  often  tends  to  inflate  these  assessments  to  avoid  interfering  with  the 
careers  of  subordinate  employees.  The  suggestions  which  follow  are  concerned  with  improving  perfor¬ 
mance  appraisal  by  providing  objective,  criterion-based  information. 

1.  The  immediate  and  second-level  supervisors  should  independently  rate  subordinate  employees 
on  critical  performance  dimensions. 

2.  Peers  should  independently  rank  coworkers  on  job  performance  characteristics,  a  process  which 
will  counteract  the  leniency  of  supervisory  ratings. 

3.  In  time,  performance  appraisals  should  accumulate  in  a  person's  record.  This  information 
should  provide  a  composite  picture  of  a  candidate’s  job  performance  (as  evaluated  by  independent 
observers)  under  various  conditions  as  the  individual  progresses  on  his/her  career  path. 

4.  The  ratings  and  rankings  should  provide  to  each  individual  information  on  strengths  as  well  as 
areas  for  self  improvement. 

5.  All  ratings  by  a  given  supervisor  can  be  averaged  so  that  ratings  can  be  evaluated  in  the  con¬ 
text  of  that  supervisor’s  tendencies  (i.e.,  excessively  lenient,  a  "hard  grader,"  or  an  objective  observer). 

Such  a  system  has  important  advantages  as  a  performance  appraisal  procedure.  Specifically,  a 
broader,  more  objective  data  base  is  available  for  each  candidate.  Evidence  shows  that  peer  rankings 
correlate  with  later  promotion  rates,  which  in  turn  reflect  the  organization's  recognition  of  past  perfor¬ 
mance.  And  the  two  supervisory  ratings  are  objective  and  realistic  when  anchored  to  other  criterion 
measures.  As  data  from  these  three  sources  are  synthesized  during  the  selector  process,  the  hiring 
official  should  be  in  a  better  position  to  choose  the  best  individual. 


5  THE  BIOGRAPHICAL  INVENTORY  AND  SKETCH 


Biographical  inventory 

A  biographical  inventory  (BI)  is  a  composite,  multiple-choice  instrument  through  which  an  indivi¬ 
dual  describes  him/herself  and  his/her  background.  The  rationale  for  such  an  instrument  is  very  sim¬ 
ple:  past  behavior,  experiences,  and  self-descriptions  can  be  used  as  indicators  of  future  performance. 
The  items  in  a  BI  typically  ask  a  wide  variety  of  questions  about  a  . person's  activities,  experiences, 
sources  of  satisfaction  and  dissatisfaction,  academic  background,  attitudes  and  interests,  values,  and 
self-descriptive  evaluations  The  scoring  of  a  BI  requires  analyzing  such  biographical  information  for  its 
relevance  to  a  given  criterion  or  to  work  performance. 

CERL  considered  the  BI  to  be  a  potentially  important  part  of  any  battery  of  instruments  used  in 
selecting  top-level  management  of  the  Corps  of  Engineers.  Therefore,  a  BI  was  developed  specifically 
for  this  study  so  that  preliminary  information  about  validating  such  an  instrument  could  be  collected. 
Although  the  data  described  in  this  report  were  collected  on  a  separate  BI,  valid  data  potentially  could 
be  obtained  simply  with  additions  to  existing  personnel  forms. 

A  number  of  studies  have  verified  that  use  of  a  BI  is  one  of  the  most  promising  ways  to  identify 
managerial  talent.  Campbell,  Dunnette,  Lawler,  and  Weick  stated  that  there  has  been  a  considerable 
amount  of  research  on  predicting  managerial  effectiveness  from  biographical  data,  but  that  the  use  of 
biographical  information  in  selecting  managers  and  other  high-level  personnel  needs  to  be  increased.1' 

In  several  studies,  Laurent  reported  the  extensive  research  done  by  Standard  Oil  of  New  Jersey  on 
early  identification  of  management  potential.16  These  studies  were  designed  to  help  answer  two  ques¬ 
tions:  how  does  one  determine  success  in  management,  and  how  does  one  identify  employees  who 
have  the  potential  to  be  successful  in  management  positions?  The  first  concern  is  the  criterion  prob¬ 
lem;  the  second  is  the  prediction  problem. 

These  studies  used  biographical  information  forms  and  found  that  in  both  the  United  States  and 
Europe,  Bis  were  significant  predictors  (validities  in  the  50s)  of  managerial  success.  To  validate  these 
Bis,  one  of  the  criteria  used  in  the  studies  was  the  Laurent  Success  Index  (LSI),  which  is  a  score 
reflecting  the  relative  degree  of  success  each  individual  has  achieved,  considering  his/her  age,  salary 
classification,  and  the  company's  salary  classification  policy.  This  index  takes  the  individual's  entire 
career  experience  into  account  but  assumes  that  a  promotion  indicates  successful  performance  at  the 
lower  job  level. 

The  success  that  Laurent  had  with  such  a  composite  instrument  led  the  investigators  of  the 
present  study  to  seek  out  a  possible  composite  against  which  the  BI  could  be  tentatively  validated.  The 
composite  measure  developed  (peer  nomination  combined  with  promotion  rate  or  salary  level  corrected 
for  experience)  was  relatively  sophisticated  (though  derived  through' a  rather  simple  procedure),  and 
could  be  used  as  a  first  step  for  validating  the  BI  instrument. 

Many  forms  of  biographical  inventories  have  been  developed  to  predict  an  individual's  job  skills 
in  industrial,  governmental,  and  academic  settings.  Criteria  examined  for  their  biographical  correlates 
have  included  leadership,  creativity,  communication  skills,  intelligence,  technical  skills,  and  artistic  per¬ 
formance.  Three  industrial  studies  have  examined  the  effectiveness  of  the  BI  in  predicting  scientific 
and  engineering  talent.17  Some  of  the  criteria  used  in  these  studies  included  measures  of  creative  per¬ 
formance  (e  g.,  number  of  publications,  number  of  patents,  and  supervisory  ratings)  and  several  aspects 
of  job  performance  other  than  creativity  (eg.,  salary  corrected  for  education  and  experience, 

"  J  P.  Campbell,  M  D.  Dunnette,  E.  F..  Lawler  III,  and  K  F.  Weick,  Managerial  Behavior  Performance  ami  pfWmrness 
(McGraw-Hill,  1970) 

l6H.  Laurent,  "Early  Identification  of  Managers,"  Management  Record,  Vol  24  (1962),  pp  33-38;  H.  Laurent.  "Research  on  the 
Identification  of  Management  Potential,"  in  3.  A.  Myers,  Jr.,  ed  ,  Predicting  Managerial  Success  (Ann  Arbor,  Ml;  Foundation  for 
Research  on  Human  Behavior,  1968);  H  Laurent,  "Cross-Cultural  Cross-Validation  of  Empirically  Validated  Tests."  Journal  <>/ 
Applied  Psychology,  Vol  54  (1970),  pp  417-423. 

p  R.  L.  Ellison,  L.  R.  James,  T.  J.  Carron,  "Prediction  of  R&D  Performance  Criterion  With  Biographical  Information,"  Journal  at 
Industrial  Psychology,  Vol  5  (1970),  pp  37-57;  R.  L.  Ellison.  L.  R.  James,  D.  G.  Fox,  and  C.  W.  Taylor.  The  Analysis  and  Predic¬ 
tion  of  Dow  Scientific  Performance  (Final  report  submitted  to  Dow  Chemical  Co.,  1968);  R,  L.  Ellison.  L  R  James.  B  W 
McDonald,  and  C.  W  Taylor,  The  Prediction  of  Scientific  and  Engineering  Performance  With  Biographical  Information  (Final  report 
submitted  to  North  American  Rockwell.  1968). 


communication  skills,  group  leadership,  breadth  of  knowledge).  The  validities  of  the  B1  scales  for 
predicting  various  criteria  were  highly  significant  and  indicated  that  this  approach  could  be  used  fruit¬ 
fully  in  predicting  the  successful  engineer. 

There  are  many  advantages  to  the  biographical  inventory  approach: 

1.  It  allows  input  from  the  individual,  rather  than  subjective  input  from  various  supervisory  lev¬ 
els. 

2.  It  permits  objective  scoring,  which  does  not  rely  on  value  judgments  of  the  information. 

3.  It  takes  a  short  time  to  administer. 

4.  It  uses  an  empirical  scoring  system  developed  for  the  Corps. 

5.  It  produces  reliable  information  because  biographical  data  -  e  g.,  birth  place,  birthday  -  do  not 
change  over  time. 

6.  It  has  demonstrated  its  validity  in  previous  studies  which  have  shown  the  approach  to  be  the 
best  single  predictor  of  future  performance. 

Pilot  Testing 

Form  V  was  the  first  BI  form  built  specifically  for  the  Corps  of  Engineers  and  was  the  basis  for 
Form  V-2,  which  was  tested  in  this  study.  For  a  description  of  the  pilot  form,  its  development,  data 
analysis,  and  biographical  correlates,  refer  to  Appendix  D. 

Description  of  the  Biographical  Items 

Of  the  150  items  in  Form  V-2,  39  were  retained  from  Form  V.  From  another  BI  (Form  0) 
administered  at  Dow  Chemical  Company,  21  items  were  used,  and  90  completely  new  items  were  writ¬ 
ten  specifically  for  this  biographical  inventory.18 

Throughout  the  construction  and  development  of  Form  V-2,  the  Civil  Service  Commission's 
(now  the  Office  of  Personnel  Management)  guidelines  for  selection  devices  were  strictly  followed.  This 
meant  that  all  items  had  to  have  either  a  conceptual  relationship  with  actual  managerial  performance  or 
a  demonstrated  statistical  validity  in  the  past. 

The  items  in  Form  V-2  can  be  assigned  to  six  content  categories  (Table  8)  plus  control  scores: 

1.  Communication 

2.  Cross-Training  Experience 

3.  Technical  Competence 


Table  8 

Item  Content  Categories  of  the  Biographical  Inventory 


I.  Communication  -  Assesses  the  experience  and  interest  in  activities  such  as  writing  (reports,  articles,  pieces  for  a 
school  newspaper),  editing,  making  speeches,  etc. 

1  Cross-Training  Experience  -  Assesses  the  breadth  of  experience  in  Corps  activities  across  different  functions  (en¬ 
gineering.  planning,  construction,  operations,  research,  etc  ),  districts  or  divisions,  branches,  etc. 

J.  Technical  Competence  -  Assesses  ihe  reported  interest  and  ability  as  an  engineer,  how  interested  the  person  is  in 
scientific  fields  other  than  engineering,  how  often  others  ask  for  help  on  engineering  problems,  etc 

4  Management  Experience  and  Performance  -  Assesses  experience  in  those  activities  that  typify  managerial  positions, 
such  as  reviewing  and  evaluating  projects,  allocating  resources,  interacting  with  other  organizations,  etc. 

5.  Innovation  --  Measures  the  reported  interest,  involvement,  and  accomplishments  in  finding  new  approaches  to  problems 
and  procedures 

6.  Management  Knowledge  -  Assesses  knowledge  of  management  principles  and  practices 


"  R.  L.  Ellison,  L  R  James,  0  G.  Fox,  and  C  W  Taylor.  The  Analysis  and  Prediction  of  Dow  Scientific  Performance  (Final  report 
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4  Management.  Experience.  and  Performance 

5  Innovation 

6  Management  Knowledge 

7  Control  Scores  Which  Include  Age,  Education.  Experience,  Minority/Nonminority,  GS 
Level,  and  Supervisor/ Nonsupervisor 

Statistical  Analysis 

All  items  in  Form  VO  were  analyzed  to  establish  their  relationships  with  certain  criteria,  including 
peer  ranking,  immediate  supervisors’  ratings,  second-level  supervisors'  ratings,  and  promotion  rate. 
Figure  6  illustrates  the  statistics  which  were  generated  for  each  item  alternative  and  each  of  the  criteria 
This  statistical  analysis  allowed  the  construction  of  keys  which  produced  scores  predicting  performance 
criteria 

Summary  of  Test  Results 

Based  on  the  pilot  data,  the  items  which  have  produced  significant  relationships  with  the 
managerial  performance  criteria  portray  the  effective  manager  as  one  who  is  experienced  and  comfort¬ 
able  in  communicating  with  people  on  a  direct  basis,  demonstrates  a  good  deal  of  flexibility  in  working 
on  several  activities  at  one  time,  is  quick-witted  and  able  to  think  under  pressure  very  well  Such  indi¬ 
viduals  describe  themselves  as  being  more  skilled  in  higher-order  managerial  activities  rather  than  in 
detailed  technical  activities.  Managers  with  high  scores  on  the  Bl  indicated  that  they  liked  to  have  a 
good  deal  of  responsibility  in  their  jobs  and  that  they  had  a  variety  of  work  experience.  These 
managers  were  interested  and  effective  in  instituting  new  policies,  procedures,  and  usually  demon¬ 
strated  greater  interpersonal  skills,  they  generally  reported  that  they  were  fairly  effective  in  pursuing 
their  managerial  duties  and  demonstrated  maturity  of  judgment  about  them. 
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Figure  6.  Example  of  item  analysis  printout  showing 
location  of  various  statistics. 
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Biographical  Sketch 

The  Corps  of  Engineers  now  uses  two  forms  for  background  information  on  the  employee:  Stan¬ 
dard  Form  171,  and  Form  2302.  The  forms  are  inconvenient  to  review  because  they  are  lengthy  and 
complex.  More  important,  neither  form  is  a  valid  description  of  an  individual’s  background,  for  neither 
contains  complete  information  on  a  candidate. 

The  proposed  Biographical  Sketch  was  developed  to  solve  these  problems;  it  provides  on  one  page 
essential  information  from  all  records  in  a  personnel  folder.  This  sketch  is  not  meant  to  replace  any 
existing  forms  but  is  a  supplemental  summary  sheet. 

Form  Description 

The  Biographical  Sketch  (Figure  7)  was  designed  to  gather  specific  details  about  the  respondent’s 
activities  and  awards.  The  12  items  obtain  information  on  age,  GS  level,  entry  level,  time  in  grade, 
length  of  service,  specific  awards,  and  activities  such  as  work  on  committees  or  special  projects.  The 
form  could  be  made  even  more  useful  with  two  additional  categories:  Suggestion  Awards  and  Profes¬ 
sional  Licensure. 

Advantages  of  the  Biographical  Sketch 

The  data  obtained  are  very  objective  and  can  be  easily  verified  from  personnel  records.  The  form 
is  easily  completed,  quickly  reviewed,  and  relevant  to  the  employee.  Such  a  brief  sketch  would  also 
have  advantages  for  the  ad  hoc  process.  Because  the  amount  of  information  for  each  candidate  is  fairly 
extensive,  the  ad  hoc  process  takes  substantial  time  from  key  people  in  an  organization;  the  procedure 
would  be  more  efficient  if  these  brief  Biographical  Sketches  were  available  to  members  of  the  commit¬ 
tee.  The  Biographical  Sketch  should  be  identified  only  with  a  Social  Security  number  to  increase  the 
objectivity  of  the  procedure. 


This  form  is  part  of  the  study  on  the  development  of  performance 
indicators  for  manaqerial  talent.  All  information  will  be  used 
for  research  purposes  only  and  will  not  be  available  to  anyone. 
Answer  the  questions  by  making  a  check  mark  opposite  your  choice 
or  by  filling  in  the  blank  where  appropriate. 


1 .  Name :  _ 

2.  What  is  your  age?  _ 

3.  What  is  your  current  GS  level?  _ 

4.  How  Iona  have  you  been  at  your  current 
GS  level? 

_  years 

5.  At  what  GS  level  did  you  enter  the 
Corps? 

6.  How  long  have  you  been  employed  by 
the  Corps? 

_  years 

7.  Have  you  won  a  Sustained  Superior 
Performance  Award? 

_ A.  No 

_ B.  Yes.  Year  of  most 

recent  award: 

8.  Have  you  won  a  Quality  Step  Increase? 

_ A.  No 

_ B.  Yes.  Year  of  most 

recent  increase: 

9.  Have  you  won  a  Special  Act  Award? 

_ A.  No 

_ B.  Yes.  Year  of  most 

recent  award: 

10.  Are  you  professionally  licensed? 

_ _A.  Yes.  Year  obtained:  _ 

B.  No 


11.  Have  you  ever  received  an 
Outstandina  Performance  Rating? 

_ _ A.  Yes.  Year  obtained: 

_ B.  No 

12.  During  the  past  three  years,  how 
often  have  you  worked  on  a 
committee,  special  project,  or 
special  assignment,  etc.,  that 
was  not  directly  concerned  with 
your  technical  work  or  specialty? 

A.  Not  at  all 

_ B.  Once 

_ C.  Twice 

_ D.  Three  times 

E.  Pour  or  more  times 


If  any,  list  the  two  most  recent: 


13.  In  your  past  work,  have  you 

voluntarily  learned  to  do  some 
new,  somewhat  difficult,  and 
relatively  burdensome  task, 
because  you  thought  it  was 
important  to  do  so? 

_ A.  Almost  never 

B.  Once  in  a  while 

_ C.  Occasionally 

_ D.  Fairly  often 


Examples: 


Figure  7.  Biographical  Sketch. 
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6  INTERVIEWING  MANUAL  AND  GUIDE 


The  interview  is  one  widely,  but  not  systematically,  used  selection  technique,  information  gained 
through  the  interview  may  not  be  available  from  any  other  source,  and  the  impressions  of  the  inter¬ 
viewer  can  make  a  valuable  contribution  to  effective  personnel  decisions.  However,  interviewing, 
though  basically  an  exchange  of  information,  is  a  complex  skill  requiring  some  sophistication  and 
experience  For  example,  if  three  people  are  being  interviewed  for  a  position,  each  interview  must  be 
conducted  similarly,  with  essentially  the  same  questions  being  asked  so  (hat  each  applicant  has  an  equal 
chance. 

Because  the  selecting  official  often  has  had  no  training  in  effective  interviewing  techniques,  an 
interview  manual  (Appendix  E)  and  guide  were  developed  to  provide  interviewers  a  framework  for  col¬ 
lecting  relevant  information.  These  tools  should  help  maintain  objectivity  and  increase  the  validity  of 
the  interview  situation.  The  guide  contains  questions  that  can  be  asked  during  the  interview;  the 
manual  provides  instructions  on  use  of  the  guide. 

The  interview  manual  and  guide  are  quite  general  and  can  fit  most  interviewing  situations  for 
managers  in  the  Corps.  However,  because  of  the  subjective  nature  of  the  personal  interview,  any  infor¬ 
mation  gathered  should  be  used  only  to  supplement  and  verify  information  obtained  from  other 
sources. 


Guide  Development 

Methods  of  interviewing  were  examined  during  the  discussions  with  Corps  employees  conducted 
for  this  study,  for  example,  how  often  were  interviews  used,  what  procedures  were  typically  followed, 
and  were  there  opportunities  for  improvement  in  this  part  of  the  promotion  and  selection  system 
Literature  on  the  hiring  process  reports  that  the  interview  is  one  of  the  most  widely  used  selection  tech¬ 
niques.1''  Invariably,  supervisors  and  employers  want  to  get  acquainted  with  the  applicant,  provide 
essential  information  about  the  job,  and  generally  size  up  the  individual  before  making  a  hiring  deci¬ 
sion.  However,  there  is  little,  if  any,  evidence  validating  the  interview  as  it  is  routinely  used  in  most 
organizations.  At  best,  the  validities,  with  only  a  few  exceptions,  tend  to  be  modest.  Interviews  typi¬ 
cally  are  most  valid  when  they  focus  on  observable  behaviors  or  on  questions  that  are  easily  interpreted 
and  scored  and  are  highly  relevant  to  critical  job  behaviors.  To  be  effective,  then,  the  interview 
requires  sophisticated  procedures,  trained  interviewers,  and/or  effective  instruments  -  in  short,  it  must 
be  carefully  prepared  and  conducted 

Given  this  general  state  of  affairs,  it  was  not  surprising  to  find  that  there  was  a  lack  of  systematic 
interviewing  procedures,  a  wide  variability  in  interviewing  skills,  and,  therefore,  room  in  the  Corps  for 
substantial  improvements  in  this  area  of  the  selection  process.  Before  the  main  data  collection  effort, 
interview  procedures  were  tentatively  explored,  but  only  a  limited  amount  of  information  was  obtained 
After  this  experience,  a  draft  of  an  interview  manual  and  guide  was  written  so  that  potential  users  of 
the  procedure  could  offer  comments.  Although  the  guide  was  only  a  preliminary  version,  suggestions 
for  improving  it  were  few.  In  general,  managers  within  the  Corps  had  not  seen  such  a  guide  before  and 
were  reluctant  to  comment  on  interview  procedures  which  were  so  unfamiliar.  Thus,  there  was  little 
involvement  and  input  from  those  asked  about  how  the  procedure  might  be  improved.  Nonetheless, 
the  interview  manual  and  guide  were  revised  with  the  information  available. 


Using  the  Interviewing  Guide 

The  Interviewing  Guide  can  be  used  by  individuals  or  in  group  interviewing  situations.  In  any 
case,  as  interviewers  receive  more  training  and  gain  experience  with  the  Guide,  the  result  will  be  more 
reliable  and  valid  information  and  a  better  selection  procedure  for  higher  level  personnel  in  the  Corps 
of  Engineers.  Some  of  the  areas  covered  in  the  Interviewing  Guide  are  listed  below 
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For  all  applicants: 

1 .  Involvement  with  special  groups,  professional  societies  and  community  organizations. 

2.  Implementation  of  directives  of  higher  level  supervisors. 

3.  Work  performance  feedback  to  subordinates. 

4.  Leadership  style  of  applicant. 

5.  Establishing  a  climate  for  innovation. 

6.  Delegating  responsibility  to  subordinates. 

7.  Approach  to  communications  between  management  and  employees. 

8.  Promotion  of  coworker  cooperation. 

9.  Unusual  or  outstanding  achievements. 

10.  Possible  strengths  and  weaknesses  in  managerial  skills. 

11.  Aspirations  and  expectations  of  the  applicant. 

For  applicants  for  GS-14  and  -15  positions  only: 

1.  Making  personnel  assignments. 

2.  Plans  for  personnel  development. 

3.  Determination  of  training  needs  of  subordinates. 

4.  Experience  in  using  the  team  approach  to  management. 

5.  Approach  to  planning. 

6.  Maintaining  high  standards  of  productivity  and  performance. 

7.  Emphasis  placed  on  each  of  the  following  activities. 

a.  Reviewing  and  evaluating  projects. 

b.  Allocating  resources. 

c.  Training  and  motivating  personnel. 

d.  Initiating  improvements  in  methods  and  operations. 

e.  Using  knowledge  and  skills  in  a  technical  capacity. 

f.  Interpersonal  interactions. 

g.  Interacting  with  other  organizations. 

h.  Scheduling  work  of  organizational  components. 

i.  Long-range  planning  and  control. 

The  Scoring  System 

The  scoring  system  is  not  complicated,  and  Corps  personnel  should  be  able  to  use  it  easily.  Here 
is  an  example  of  how  an  individual  question  could  be  scored. 
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Q:  How  much  self-initiated  training  that  is  nongovernment 
sponsored  have  you  had  since  you  left  college? 

_ Technical 

_ Management,  general 

_ Management,  specific  skills 

_ Social  science 

__  Other 

Total _ 

On  this  question,  the  applicant  would  respond  with  the  kinds  of  self-initiated  training  that  he/she  has 
had,  and  the  interviewer  would  make  a  check  beside  each  type  of  training.  The  score  for  the  applicant 
on  this  question  would  range  from  0  (no  training)  to  5  (training  in  each  category). 

The  basic  advantage  of  this  scoring  system  is  that  it  gives  applicants  who  have  had  diverse  training 
or  experiences  more  credit  than  those  who  have  not.  The  system  would  make  the  interview  more 
objective  and  more  standardized  across  applicants;  thus,  the  interview  could  become  a  more  valid,  and 
therefore  useful,  part  of  the  selection  process. 


The  Interview’s  Place  in  the  Hiring  Process 

An  interview  should  not  be  the  critical  determinant  in  any  hiring  process  -  it  should  be  con¬ 
sidered  only  one  source  of  information.  Since  the  interview  will  inevitably  be  used  at  least  in  selected 
situations,  improving  that  source  of  information  and  making  the  information  obtained  more  relevant, 
accurate,  and  useful  in  hiring  decisions  is  essential.  But  the  most  effective  selection  and  promotion 
decisions  will  result  from  considering  information  from  a  number  of  sources. 


7  THE  MANAGEMENT  AUDIT  SURVEY 


The  Management  Audit  Survey  (MAS)  involves  all  levels  of  management  in  identifying  and  over¬ 
coming  problem  areas  in  the  work  environment  to  stimulate  an  organization-wide  movement  toward 
improved  management  practices,  increased  productivity,  and  higher  levels  of  employee  satisfaction 
The  MAS  system  -  initially  developed  for  the  Department  of  Labor  -  has  been  under  refinement  for 
nearly  10  years,  its  potential  use  to  the  Corps  of  Engineers  was  first  discussed  in  1977. 20  Now,  the 
feasibility  study  to  determine  whether  the  MAS  would  be  useful  for  the  Corps  has  been  completed. 
This  chapter  describes  the  information  provided  by  the  system,  the  implementation  of  the  system,  and 
the  preliminary  results  obtained  after  administering  the  MAS  at  three  U  S.  Army  Corps  of  Engineers 
organizations. 

The  development  of  the  MAS  system  was  based  on  extensive  research  with  Federal  employees  in 
the  Department  of  Labor  and  the  Treasury  Department,  and  employees  from  the  private  sector 21  This 
research  demonstrated  that  the  MAS  was  reliable  both  in  consistency  within  a  score  area  and  in  the 
agreement  among  employees  within  a  work  group  on  their  description  of  their  work  environment.  The 
two  studies  by  Ellison,  et  al.,  provided  evidence  that  the  MAS  scores  predicted  organizational  perfor¬ 
mance  criteria  relevant  to  human  resource  management. 

The  MAS  system  is  a  combination  of  a  100-item,  employee-completed  questionnaire,  a  computer¬ 
ized  method  for  providing  feedback  to  supervisors  (Appendix  F),  a  handbook  to  assist  supervisors  in 
developing  programs  for  constructive  change,  and  individual  action  plans  to  be  completed  by  each 
manager  or  supervisor  who  receives  feedback  (Appendix  G).  The  MAS  provides  quantified  indications 
of  managerial  performance  and  organizational  characteristics  for  areas  which  formerly  could  be  assessed 
only  in  a  limited  or  subjective  manner.  To  avoid  subjective  assessments,  the  MAS  was  designed  with 
an  underlying  behavioral  emphasis.  This  emphasis  aids  development  of  programs  for  constructive 
change  because  the  questionnaire  itself  contains  descriptions  of  those  activities  which  can  lead  to 
improvement  of  a  particular  management  operation.  Thus,  the  system  provides  both  the  impetus  for  a 
review  of  current  management  practices  and  an  opportunity  to  revitalize  the  organization. 


How  the  MAS  System  Works 

With  the  MAS  system,  each  supervisor  with  five  or  more  employees  responding  to  the  survey 
form  receives  a  computer-generated,  diagnostic  description  of  management  practices  and  the  work 
environment  as  defined  by  anonymous  employee  responses  to  the  100-item  questionnaire.  Higher  level 
supervisors  receive  two  reports,  one  based  on  the  responses  of  the  employees  in  the  immediate  work 
group  and  a  second,  consolidated  report  based  on  all  subordinate  employees.  The  system  covers  19  key 
score  areas  relevant  to  human  resource  management  and  the  quality  of  working  life  A  brief  descrip¬ 
tion  of  each  of  the  MAS  scores  is  presented  in  Table  9.  The  score  areas  cover  assets  of  managerial 
performance  such  as  Fairness  of  Management,  Delegation  of  Authority,  and  Supervisory  Effectiveness, 
assessments  of  program  effectiveness  including  Equal  Opportunity  for  Minorities  and  Women,  and 
areas  more  directly  focused  on  the  employee,  such  as  Morale,  Work  Satisfaction.  Opportunity  for  Pro¬ 
motions,  and  Satisfaction  with  Pay 

The  employees  respond  to  the  MAS  items  on  answer  sheets  computer-scored  by  the  Institute  for 
Behavioral  Research  in  Creativity,  which  developed  the  system.  This  procedure  provides  the  employee 
a  further  degree  of  anonymity  because  no  one  in  the  surveyed  organization  -  other  than  the  respon¬ 
dent  --  sees  the  answer  sheet  As  part  of  the  processing,  all  the  answer  sheets  are  screened  by  a  spe¬ 
cially  developed  computer  program  (SPECTR)  to  remove  distorted  data;  i.e.,  answer  sheets  that  are  too 
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Table  9 


Description  of  the  19  MAS  Scores 


fairness  of  Management  measures  how  fairly  employees  are 
treated  by  management,  whether  fairness  governs  promotions  and 
other  10b  functions,  and  if  adequate  credit  is  given  for  work 
well  done 

Delegation  of  Authority  is  a  measure  of  the  degree  of  definition  of 
lines  of  authority  and  responsibility,  including  whether 
sufficient  authority  is  given  employees  to  plan  their  work 
independently 

Supervisory  Effectiveness  measures  the  overall  effectiveness  of 
supers  ision.  the  supervisor's  specific  influence  on 
the  performance  of  employees  in  the  unit,  and  the 
quality  of  supervisory  review  and  follow-up 

Planning  and  Administrative  Efficiency  measures  the  overall 
effectiveness  of  planning  and  the  level  of  efficiency 
within  a  unit  Amount  of  unnecessary  paperwork,  amount 
of  time  wasted  because  of  poor  planning,  and  rules  and 
procedures  that  facilitate  or  inhibit  performance  are 
all  a  part  of  this  score  area 

Climate  for  Innovation  reflects  the  number  of  discussions  held  on 
new  ideas,  methods  and  ideas,  the  manner  of  receiving  new  ideas 
and  the  follow-up  treatment  of  suggestions  for  methods  or 
approaches  to  the  work  It  is  a  measure  of  the  general 
openness  to  new  ideas  and  methods 

Work  Satisfaction  measures  the  degree  of  interest  and  pleasure 
offered  by  the  work,  the  opportunity  the  work  provides 
for  accomplishment,  and  the  degree  of  satisfaction  obtained 
from  the  work  itself 

Training  Effectiveness  measures  the  effectiveness  of  on-the-job 
training,  new  employee  training,  and  training  for  new 
work  methods  and  procedures  Effectiveness  in  recognizing 
training  needs  is  also  measured 

Performance  feedback  is  a  measure  of  the  quantity  and  quality  of 
the  work  performance  discussions  held  with  employees. 

It  assesses  the  number  of  times  work  performance  is 
discussed  with  an  employee,  the  amount  of  useful  information 
that  is  conveyed,  and  the  employee's  reaction  to  these 
discussions 

Equal  Employment  Opportunity  for  Women  assesses  whether  women  have 
the  same  opportunity  as  men  to  be  hired  and  promoted, 
to  receive  training,  to  become  supervisors,  and  evaluates  the  general 
level  of  success  of  equal  opportunity  programs  for  women 
in  the  organization 

Equal  Employment  Opportunity  for  Minorities  assesses  the  same 
dimensions  of  employment  opportunity  as  those  measured 
by  number  A  above,  with  the  focus  on  minority  group  members 


Table  9  (Cont’d) 


1 1  Opportunity  lor  Promotion  measures  promotion  opportunities  within 
and  upward  from  the  work  group,  to  specific  lines  of  work 

and  whether  existing  opportunities  encourage  employees  to  stay 
with  the  unit  and  work  hard  for  promotion. 

12  Downward  Communication  assesses  effectiveness  of  downward 
communication  by  examining  whether  employees  know  what 

results  are  expected  from  work  assignments,  whether  the  advance  notice 

is  given  of  changes,  whether  explanations  of  changes  are 

given,  and  whether  the  employees  are  generally  kept  well-informed 

13  Upward  Communication  measures  management’s  willingness  to  consider 
ideas  and  problems  of  lower  level  employees.  It  measures 

the  degree  of  interest  in  and  awareness  of  employee  attitudes, 
problems,  and  ideas 

14  Satisfaction  With  Pay  is  a  measure  of  perceived  adequacy  of  pay  in 
relation  to  assigned  work  and  compared  to  other  similar  jobs 

and  the  local  cost  of  living. 

15  Morale  measures  whether  employees  compare  their  work  group 
favorably  with  other  groups  as  a  place  to  work,  see 
themselves  as  having  a  good  future  in  the  organization,  and 
would  stay  with  the  unit  if  offered  a  similar  job  elsewhere  at 
the  same  pay 

16  Physical  Working  Conditions  and  equipment  is  a  measure  of  the 
quality  of  light,  heat,  air,  equipment,  supplies,  work 

space,  furniture,  and  cleanliness  of  restrooms  and 
and  other  facilities. 

17.  Co-worker  Cooperation  assesses  the  amount  of  free  information 
exchange  between  workers,  the  willingness  of  co-workers 
to  assist  each  other,  their  ability  to  work  together  toward 
unit  objectives,  and  the  amount  of  unpleasant  disagreement  in 
the  group 

18  Operational  Efficiency  measures  how  well  the  group  handles 
problems,  difficult  projects,  and  pressure,  as  well  as 
meeting  objectives  and  adapting  new  approaches  to  problems 

19.  Workload  Balance  measures  the  balance  between  the  work  to  be  done 
and  the  staff  assigned  to  the  work.  This  includes 
both  over-  and  under-staffing 


incomplete,  that  are  completed  randomly,  that  are  overwhelmingly  positive  or  negative,  or  that  vary  lit¬ 
tle  in  their  descriptions  of  organizational  procedures  The  program  helps  increase  the  accuracy  and 
meaningfulness  of  the  feedback  reports 

The  system  provides  two  forms  of  feedback  on  the  19  score  areas: 

1.  A  percentile  rank  score  compares  the  average  scores  of  employees  within  a  work  group  to  the 
overall  organization's  average  or  to  a  major  organizational  component’s  average.  With  this  procedure,  a 
score  above  the  50th  percentile  is  above  average.  For  example,  a  percentile  rank  of  70  means  that  the 
average  score  of  the  people  in  the  work  group  exceeded  70  percent  of  the  scores  obtained  in  the  entire 
organization. 

2.  A  percent  favorable  score  is  based  on  how  often  employees  in  the  total  organization  and  in 
large  organizational  components  used  favorable  response  options  to  answer  the  questions  in  each  score 
area.  This  absolute  scce  allows  comparisons  of  large  organizational  components  across  scores  to 
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provide  information  about  areas  most  needing  attention  throughout  the  organization,  eg.  fairness  of 
Management  and  Co-worker  Cooperation  may  be  typically  viewed  more  favorably  than  Performance 
Feedhack  and  Opportunity  for  Promotions. 

With  the  feedback  report,  the  supervisor  also  receives  the  Handbook  for  Supervisors,  a  47-page 
booklet  explaining  the  system  It  includes  an  explanation  of  the  two  scoring  procedures,  a  presentation 
of  a  sample  feedback  report,  a  description  of  each  score  area  (this  includes  a  score  definition,  key  score 
interrelationships,  a  paragraph  describing  examples  of  outside  factors  that  may  affect  the  score  areas, 
and  behavioral  suggestions  for  improving  score  performance),  and  an  appendix  which  presents  the 
items  that  are  used  to  assess  each  score  area. 

Based  on  the  feedback  report  about  the  relative  strengths  and  weaknesses  of  management  prac¬ 
tices  as  seen  by  employees,  the  information  provided  in  the  handbook,  and  his  or  her  own  knowledge 
of  the  work  environment,  the  supervisor  -  with  the  aid  of  subordinate  employees  -  formulates  an 
Action  Plan  designed  to  bring  about  higher  levels  of  performance  in  those  problem  areas  identified  by 
the  MAS  system.  The  Action  Plan  should  include  six  key  steps  in  initiating  and  planning  organizational 
change.  These  steps  are: 

1.  The  definition  of  a  problem  area. 

2.  A  decision  concerning  the  use  of  a  task  group  to  gather  additional  information  and  make 
recommendations. 

3.  A  statement  of  new  procedures,  behaviors,  and  policies  to  be  followed. 

4.  A  statement  of  how  the  plan  is  to  be  followed  up  with  feedback  to  insure  the  new  activities  are 
solving  the  problem(s)  identified. 

5.  A  designation  of  who  has  responsibility  for  the  change  process. 

6  A  statement  concerning  what  resource  people  (personnel,  training,  management  analysis),  if 
any.  are  to  be  used  to  help  resolve  the  problems. 

Typically,  the  two  or  three  score  areas  selected  for  action  are  those  which  are  relatively  low  and 
which  the  supervisor  can  change.  The  Action  Plans  are  discussed  with  the  next  higher  level  of  supervi¬ 
sion  which  does  not  have  access  to  the  actual  score  results  but  can  stimulate  effective  problem  solving 
at  lower  levels  The  system  is  thus  designed  to  initiate  problem  solving  activities  throughout  the  organ¬ 
ization  in  a  nonthreatening  manner.  Since  the  system  is  designed  to  be  acted  upon  by  line  manage¬ 
ment,  it  provides  a  means  for  an  organization,  through  the  efforts  of  the  employees,  to  assess  and 
improve  managerial  performance,  and  thus  achieve  greater  productivity  and  worker  satisfaction. 

Since  the  system  was  designed  to  be  run  at  12-  to  24-month  intervals,  other  procedures  and  rein¬ 
forcement  mechanisms  need  to  be  integrated  with  the  MAS  to  insure  that  the  system  operates  at  an 
optimal  level  These  may  include  training  programs  on  management  areas  needing  attention,  follow-up 
of  Action  Plans,  and  the  use  of  management  teams  to  investigate  new  procedures.  Effective  use  of  the 
MAS  and  other  innovative  strategies  requires  a  significant  commitment  by  management  -  i  e  .  a  real 
willingness  to  work  toward  innovative  procedures  to  improve  levels  of  performance  and  employee  satis¬ 
faction. 


Objectives  of  the  MAS  Study 

The  study  reported  in  this  chapter  was  conducted  to  acquaint  Corps  managers  with  the  system  and 
to  examine 

1  Diagnostic  capabilities  of  the  system  in  analyzing  management  practices  at  the  work  group  and 
the  organizational  level. 

2.  Ways  the  system  could  strengthen  management  capabilities  and  institute  a  process  of  organiza¬ 
tional  change. 

3  Effectiveness  of  management  in  dealing  with  the  information  provided  by  the  system. 
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Procedures 


Preliminary  Arrangements 

Three  Corps  field  offices  (a  Division  office,  Dislrici  office,  and  research  laboratory)  were  visited  to 
solicit  their  cooperation  and  to  discuss  the  nature  of  the  project.  All  three  sites  agreed  to  participate. 
An  important  point  stressed  about  the  study  was  that  the  survey  results  were  to  be  used  as  a  construc¬ 
tive  management  tool  to  initiate  change  within  the  organization  and  not  to  induce  threat  or  anxiety, 
which  would  limit  the  effectiveness  of  the  system.  Without  exception,  all  the  Chief  Executive  Officers 
of  the  three  sites  agreed  to  this  constructive  approach  -  operationally  defined  by  the  strategy  of  allow¬ 
ing  no  one  to  see  a  copy  of  a  feedback  report  other  than  the  person  to  whom  that  report  applied  This 
technique  suggested  that  lower  level  supervisors  were  trusted  to  solve  the  problems  that  existed. 

Also  during  the  preliminary  visit,  one  or  two  people  were  selected  to  serve  as  facilitators.  These 
individuals  were  to  perform  a  very  important  role,  they  would  coordinate  the  administration  of  the 
MAS  and  be  largely  responsible  for  the  feedback  of  the  results.  In  addition,  they  would  act  as  resource 
individuals  to  help  supervisors  realize  opportunities  for  improvement  from  the  feedback  reports.  The 
facilitators  were  briefed  through  telephone  conversations,  and  a  handbook  was  sent  to  them  in  prepara¬ 
tion  for  a  special  two-day  facilitator’s  workshop  held  at  a  central  location  to  train  individuals  from  each 
site. 

Administrative  Arrangements 

After  the  workshop,  the  facilitators  assumed  most  responsibility  for  administering  the  survey. 
This  procedure  was  followed  to  reduce  costs  and  to  begin  a  process  through  which  individuals  at  the 
sites  became  involved  in  insuring  that  meaningful  results  were  obtained.  The  administrative  procedures 
in  general  were  effective  --  very  high  percentages  (89  percent  to  99  percent)  of  the  employees  partici¬ 
pated  at  the  three  sites. 

The  individual  answer  sheets  were  sent  directly  to  the  contractor  for  scoring,  which  was  completed 
approximately  2  weeks  after  receipt  of  the  employees'  responses.  The  survey  results  were  then  ready 
for  dissemination  to  the  individual  sites. 

Feedback  oj  the  Survey  Results 

The  feedback  strategies  used  at  each  of  the  three  sites  varied  to  meet  local  circumstances.  What 
follows  is  a  general  picture  of  the  feedback  process,  with  occasional  comments  to  illustrate  how  pro¬ 
cedures  were  modified  for  each  site 

An  appointment  was  made  with  the  Chief  Executive  Officer  approximately  1  week  before  the 
feedback  was  ready  for  distribution.  Upon  arrival  at  the  particular  site,  and  after  a  preliminary  discus¬ 
sion  with  the  facilitators,  the  contractor’s  representative  met  with  the  Chief  Executive  Officer,  a  few  of 
his  immediate  assistants,  and  the  facilitators.  After  a  discussion  of  the  system,  the  contractor’s 
representative  presented  to  the  Chief  Executive  Officer  the  percent  favorable  results  on  the  site  as  a 
whole.  These  results  were  compared  to  data  on  over  30.000  individuals  in  18  other  organizations  and 
the  other  two  Corps  sites.  In  this  manner,  the  Chief  Executive  Officer  had  an  opportunity  to  review  the 
overall  data  in  private  to  become  familiar  with  it  and  its  implications.  At  this  point  the  data  were 
"owned’’  by  the  site  -  not  by  the  contractor  An  agenda  was  then  worked  out  for  presenting  these  data 
to  ail  individuals  who  reported  immediately  to  the  Chief  Executive  Officer  -  typically.  Division  Chiefs 
and  Office  Chiefs.  These  individuals  were  given  the  survey  results  for  the  entire  organization  and  told 
that  the  facilitator  (and  the  contractor’s  representative  during  that  week)  would  meet  with  them 
privately  both  to  return  their  own  results  and  to  plan  a  meeting  for  reporting  those  results  to  their 
immediate  employees.  In  this  way  the  data  were  presented  to  the  entire  organization,  with  each  super¬ 
visor  having  an  opportunity  for  essentially  three  briefings  one  from  his/her  supervisor  about  the 
results  at  that  level,  one  with  the  facilitator  and  contractor  representative  to  insure  that  the  results  were 
interpreted  correctly  at  the  next  lower  level,  and  finally  a  discussion  with  lower  level  employees  to 
disseminate  the  results  and  plan  the  next  steps. 

In  all  the  preliminary  feedback  meetings  between  the  Chief  Executive  Officer  and  the  facilitators, 
one  of  the  topics  mentioned  was  the  importance  of  having  the  supervisor  or  manager  lead  the  feedback 
session  presenting  survey  results  to  his/her  subordinate  employees.  This  process  substantially  began 
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the  transfer  of  "ownership"  of  the  data;  i.e.,  the  data  no  longer  belonged  to  the  research  organization 
which  developed  the  survey  and  processed  the  responses,  but  to  the  individual  manager  who  knew  the 
situation,  had  the  responsibility  for  initiating  change,  and  was  in  a  better  position  than  anyone  else  to 
understand  and  act  upon  the  implications  of  the  data.  After  some  brief  discussion  of  strategies  that 
could  be  used  during  a  feedback  meeting  with  subordinates,  most  supervisors  and  managers  accepted 
their  role  with  little  difficulty,  needing  only  occasional  assistance  from  the  facilitators  and  consultant. 

Obviously,  the  facilitators  played  an  important  role  in  the  feedback  of  the  survey  results  because 
they.  (1)  were  the  on-site  representatives  of  the  survey,  (2)  were  knowledgeable  about  the  procedures, 
(3)  had  the  support  of  the  Chief  Executive  Officer  in  seeing  that  the  survey  results  were  used  construc¬ 
tively,  and  (4)  were  in  a  position  to  marshal  resources  to  assist  people  with  difficulties.  However,  one 
of  the  problems  encountered  at  all  three  sites  was  the  conflict  between  the  ongoing  job  duties  of  the 
facilitators  and  the  activities  required  by  the  MAS  system. 


Results 

This  section  presents  the  results  obtained  to  date  on  a  wide  range  of  topics  and  includes  an  exam¬ 
ple  of  a  feedback  report  with  suggested  strategies  for  change,  a  discussion  of  Action  Plans,  a  description 
of  the  relative  standing  of  the  three  Corps  locations,  selected  results  obtained  with  individual  MAS 
items,  and  the  results  of  follow-up  visits  to  the  selected  locations. 

Example  f  eedback  Report 

Table  10  is  an  example  of  a  unit  which  is  typical  in  that  some  scores  are  high  and  others  are  low. 
The  table  compares  the  unit  of  eight  employees  to  the  total  Corps  Sample  in  the  first  column  (Organi¬ 
zation)  and  to  the  rest  of  the  local  site  -  District,  Division,  or  laboratory  -  in  the  second  column 
(Component).  However,  as  the  table  illustrates,  the  unit  does  have  some  serious  problems.  Of  the  19 
MAS  percentile  rank  scores,  13  are  in  a  range  (below  40)  which  would  indicate  that  difficulties  probably 
exist.  Of  these  13,  six  are  below  30  -  definitely  indicating  a  need  for  improvement.  A  percentile  rank 
of  30  indicates  that  the  average  score  for  the  unit  surpassed  only  30  percent  of  the  Corps  employees 
surveyed,  or  70  percent  of  the  Corps  employees  surveyed  described  their  work  environment  more  posi¬ 
tively  than  the  average  obtained  in  that  unit. 

Of  particular  interest  in  Table  10  is  the  percentile  rank  for  Delegation  of  Authority.  70.  This  is  a 
relatively  high  score  indicating  that  the  average  for  the  unit  exceeded  the  raw  scores  of  70  percent  of 
the  Corps  employees  surveyed.  Apparently,  employees  were  given  a  free  hand  to  plan  their  work 
independently.  However,  this  freedom  evidently  has  not  been  directed  effectively,  nor  is  there  evi¬ 
dence  that  standards  were  imposed  on  the  quality  and  quantity  of  work  output.  The  low  scores  on 
Supervisory  Effectiveness  and  Planning  and  Administrative  Efficiency  indicate  inadequate  review  and 
follow-up  of  work  done,  poor  planning  and  scheduling  of  how  work  was  to  be  done,  and  little  help  from 
the  supervisor  in  solving  work  problems.  A  good  deal  of  time  apparently  was  spent  doing  unnecessary 
paperwork.  The  Communication  scores,  though  not  extremely  low,  do  indicate  employees  perceived 
that  management  lacks  interest  in  and  awareness  of  subordinates’  attitudes,  problems,  and  ideas  for 
dealing  with  problems. 

Other  MAS  scores  indicate  that  the  employees  were  close  to  the  norm  in  feeling  that  they  could 
get  ahead  in  the  work  unit  or  organization,  in  Satisfaction  with  Pay,  and  in  feeling  that  management 
treated  employees  equitably.  The  relatively  low  scores  on  Morale,  Co-worker  Cooperation,  and  Work 
Satisfaction  probably  reflect  the  problems  manifested  in  other  scores.  The  fairly  low  score  on  Opera¬ 
tional  Efficiency  (percentile  rank  =  27)  is  a  further  overall  indicator  of  the  unit's  level  of  performance 
The  employees  saw  some  problems  in  the  group’s  ability  to  produce  quality  work;  this  perception  might 
include  difficulties  in  handling  projects,  solving  problems  that  arise,  working  under  pressure,  meeting 
objectives,  and  adopting  new,  more  effective  approaches  to  problems. 

In  developing  an  Action  Plan  for  this  unit,  the  supervisor  should  recognize  two  factors;  (1)  there 
aie  strengths  that  could  be  used  as  building  blocks,  and  (2)  only  two  or  three  of  the  lower  scores 
should  be  chosen  for  Action  Plan  strategies  since  an  attempt  to  deal  with  all  the  low  scores  would  be 
too  diluted.  The  management  areas  selected  should  be  those  that  have  the  most  impact  on  the  unit's 
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Table  10 


Feedback  Report  of  a  Typical  Unit* 

MAS  Results  for  ll. S.  Arm)’  Corps  of  Engineers 
Organization  Codes  Which  Begin  With _ 

8  Employees  Included 

Score 

1  I  airness  of  Management 

2  Delegation  of  Authority . . 

3.  Supervisory  Effectiveness . 

4  Planning  &  Administrative  Efficiency  . 

5  Climate  lor  Innovation . . 

b  Work  Satisfaction  . 

7.  Training  Effectiveness . 

8.  Performance  Feedback . 

9  Equal  Opportunity  for  Women . 

10  Equal  Opportunity  for  Minorities . . . 

1 1  Opportunity  for  Promotions . 

12  Downward  C  ommunication . 

i  t  I  Ipward  (  ommunication . . .  . 

14  Satisfaction  with  Pay . 

15  Morale  . . 

16  Physical  Working  Conditions  &  Equip . 

1 7.  Co-Worker  Cooperation . 

18  Operational  Efficiency . 

19.  Workload  Balance . 

66%  Understaffing  Responses 
3%  Overstaffing  Responses 


Percentile  Rank  of  l  nit  Score 
In  Relation  to  Scores  of: 


Organization 

Component 

41 

41 

70 

i.t 

26 

30 

17 

21 

32 

35 

31 

32 

36 

40 

35 

37 

27 

30 

29 

33 

59 

5b 

4b  4' 

31  .13 


58 

bl 

30 

14 

2b 

31 

38 

43 

27 

.19 

42 

53 

'Coding  information  and  percent  favorable  results  were  omitted  to 
maintain  anonymity  of  the  unit 
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performance.  1  his  unit's  efficiency  should  be  improved  rather  easily,  assuming  that  management  is  wil¬ 
ling  to  realize  and  accept  that  opportunities  for  improvement  exist. 

Action  Plans 

To  effect  organizational  change,  the  behavior  of  individuals  within  the  organization  must  change 
in  some  way.  The  MAS  system  provides  a  mechanism  -  Action  Plans  --  to  facilitate  such  change. 
Briefly,  the  supervisor  chooses  no  more  than  two  or  three  MAS  scores  which  he  or  she  feels  can  be 
improved.  By  communicating  with  employees  and  marshalling  all  potential  resources  in  the  organiza¬ 
tion.  the  manager  develops  strategies  to  solve  the  identified  problem  areas.  Table  11  is  an  actual  Action 
Plan  completed  at  a  Corps  location.  One  criticism  of  the  Action  Plans  developed  thus  far  at  the  Corps 
locations  is  that  they  fail  both  to  use  task  groups  within  the  unit  and  to  draw  upon  resource  people  out¬ 
side  the  unit.  Of  28  Action  Plans  from  one  location,  only  five  reported  the  use  of  task  groups,  and  only 
12  reported  using  resource  people  to  help  solve  one  or  more  of  the  identified  problems  on  the  plan. 
More  use  of  such  assistance  should  be  encouraged. 

The  Corps  of  Engineers  Compared  With  Other  Organizations 

Over  30,000  employees  in  a  number  of  organizations  have  responded  to  the  MAS,  so  the  Corps’ 
results  can  be  compared  with  those  obtained  elsewhere.  Table  12  presents  the  percent  favorable  results 
from  the  three  organizations  studied  within  the  Corps,  and  the  highs  and  lows  from  18  other  organiza¬ 
tions.  To  preserve  their  anonymity,  the  three  Corps  locations  have  been  rank-ordered  for  each  score 
area  with  the  lowest  of  the  three  on  the  left.  The  organization  that  is  high  on  one  score  could  on  the 
next  be  lowest,  or  the  middle,  or  high  again. 

A  comparison  of  the  three  Corps  locations  indicates  that  each  Corps  site  had  definite  strengths 
(high  scores)  on  which  management  could  build,  and  definite  weaknesses  (low  scores)  which  need  to 
be  corrected.  One  location  was  the  highest  on  four  of  the  19  MAS  scores  and  the  lowest  on  10 
Another  location  was  the  highest  on  four  scores,  and  was  the  lowest  on  two.  The  last  location  was  the 

Table  11 

Example  Action  Plan 


(1) 

(2) 

(3) 

(4) 

15) 

(6) 

Definition  of  Problem  and 
Related  MAS  Score(s) 

Task 

Group 

Formed 

Problem-Solving  Activities 

Feedback 

Planned 

Responsi¬ 

bility 

Resource 

People 

4  Planning  and  Administration  Hffic  No  Change  staff  meetings  and  No  formal  Me/DD  IX'-M  advice 

f  avorable  response  41%  --  SPR  items  of  interest  Keep  method 

mission  assumption  wilt  info  on  SPR  as  visible  as 

exacerbate  possible  Ciet  SPR  auth  and 

dollars  for  front  end 
staffing 


1 1  Opportunity  for  Promotions  No  Continue  Executive  Office  No  formal  Me/ Dl)  Personnel  as 

f  avorable  responses  only  21%  staffing  reorganization.  Support  method.  requested 

Perception  by  troops  needs  lateral  moves  for  improve- 

improvcmcni  ment.  SPR  mission  will 

provide  some  relief  -  if 
done  right 


I1)  Workload  Balance  Favorable  No 
Response  39%.  Not  as  bad  as 
they  perceive 


Change  staff  meetings  and  No  formal  Me/DD  DC-M  advice 

items  of  interest  to  better  method 

show  what's  really  going  on 


Table  12 


Management  Audit  Survey:  Percent  of  Favorable  Responses  in  the 
Corps  of  Engineers  and  in  18  Other  Organizations  Surveyed 


Other 

Corps  of  Engineers  Organizations 

Locations  Surveyed 


MAS  Score  Areas: 

Low 

Mid 

High 

Lowest 

Highest 

l 

Fairness  of  Management 

65 

65 

68 

51 

71 

2. 

Delegation  of  Authority 

50 

57 

65 

42 

61 

3 

Supervisory  Effectiveness 

51 

59 

59 

47 

67 

4, 

Planning  &  Administrative  Efficiency 

25 

37 

41 

35 

48 

5. 

Climate  for  Innovation 

47 

47 

49 

32 

50 

6. 

Work  Satisfaction 

64 

68 

71 

53 

79 

7. 

Training  Effectiveness 

42 

43 

44 

37 

58 

8. 

Performance  Feedback 

29 

30 

31 

33 

48 

V 

Equal  Opportunity  for  Women 

58 

61 

64 

49 

73 

10 

Equal  Opportunity  for  Minorities 

67 

67 

72 

47 

72 

II 

Opportunity  for  Promotions 

20 

21 

24 

19 

36 

12 

Downward  Communication 

56 

56 

57 

47 

67 

13 

Cpward  Communication 

41 

43 

45 

32 

52 

14 

Satisfaction  with  Pay 

32 

36 

41 

25 

44 

IS 

Morale 

38 

48 

53 

29 

62 

16 

Physical  Working  Conditions 

51 

57 

60 

33 

59 

17 

Co-Worker  Cooperation 

69 

70 

76 

57 

76 

18 

Operational  Efficiency 

72 

73 

81 

56 

75 

19. 

Workload  Balance 

29 

39 

46 

37 

55 

highest  on  four  scores,  tied  for  the  high  on  one.  The  last  location  was  the  highest  on  four  scores,  ami 
the  lowest  on  three. 

On  the  whole,  results  from  the  three  Corps  sites  --  unlike  those  from  other  organizations  sur¬ 
veyed  -  display  similar  patterns.  This  finding  supports  the  position  that  management  ability  across 
different  Corps  sites  is  generally  uniform.  Examples  of  scores  which  were  similar  at  the  three  sites 
include  Fairness  of  Management,  Climate  for  innovation.  Training  Effectiveness,  Performance  Feed¬ 
back,  Opportunity  for  Promotions,  and  the  Communication  measures. 

However,  on  other  scores  there  were  highly  significant  differences  among  the  three  sites  For 
example,  on  Delegation  of  Authority  (item  2)  the  average  percentage  of  favorable  scores  ranged  from  a 
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low  of  50  to  a  high  of  65.  Morale  and  Workload  Balance  also  varied  widely  across  the  three  Corps  sites, 
ranging  from  38  to  53  for  Morale  and  from  29  to  46  for  Workload  Balance. 

On  item  4.  Planning  and  Administrative  Efficiency,  there  were  also  real  differences  among  the 
organizations  studied;  the  respondents  at  one  of  the  Corps  locations  chose  only  25  percent  of  the  possi¬ 
ble  favorable  response  alternatives.  This  was  lower  than  the  scores  obtained  at  any  of  the  other  18 
organizations  studied.  The  second  lowest  Corps  facility  on  Planning  and  Administrative  Efficiency  was 
ilso  low  in  relation  to  the  other  organizations  studied. 

The  most  crucial  finding  in  the  table  is  item  8  -  Performance  Feedback.  On  this  measure, 
employees  at  all  three  Corps  sites  reported  that  they  received  only  a  limited  amount  of  helpful  informa¬ 
tion  about  their  job  performance.  In  other  organizations,  the  percentage  of  favorable  response  alterna¬ 
tives  endorsed  ranged  from  33  to  48.  That  is.  from  one  third  to  one  half  of  these  employees  described 
as  positive  and  helpful  the  quality  and  quantity  of  information  they  received  about  job  performance. 
These  findings  contrast  clearly  with  the  results  obtained  at  all  three  Corps  locations,  where  only  29,  30, 
and  31  percent  of  the  favorable  response  alternatives  were  endorsed.  Since  most  of  the  other  organiza¬ 
tions  surveyed  were  Federal  or  State  government  white  collar  organizations,  there  are  no  apparent  rea¬ 
sons  for  three  Corps  locations  to  set  new  lows.  This  is  evidently  an  issue  area  that  deserves  some 
attention.  Perhaps  the  new  performance  appraisal  program  -  which  is  being  revised  by  the  Department 
of  the  Army  and  is  now  being  examined  for  implementation  throughout  the  Corps  -  will  affect  this 
area.  The  MAS  results  of  this  new  program  and  any  other  information  available  should  be  monitored 
on  a  continuing  basis  to  evaluate  both  the  extent  and  nature  of  this  problem,  and  any  progress  in  over¬ 
coming  it 

On  item  10,  Equal  Opportunity  for  Minorities,  one  of  the  three  Corps  locations  had  a  score  which 
equaled  the  highest  obtained  in  any  of  the  other  organizations  studied.  At  this  site,  an  average  of  72 
percent  of  the  possible  favorable  response  alternatives  were  endorsed  by  employees  describing  the  EEO 
program  for  minorities.  The  other  two  locations  were  also  relatively  high  on  this  score.  However, 
minorities  typically  do  not  feel  as  positive  as  nonminorities  about  the  progress  that  has  been  made. 

On  the  Opportunity  for  Promotions  score  (item  II)  all  three  locations  scored  relatively  low. 
approximating  the  lowest  scores  from  surveys  of  other  organizations.  Since  the  survey  results  in  other 
organizations  were  obtained  two  or  more  years  before  the  Corps’  results,  these  findings  may  reflect 
some  of  the  current  restraints  on  grade  levels. 

The  three  sites  were  generally  similar  on  MAS  scores  of  Downward  Communication,  Upward 
C  ommunication,  and  Satisfaction  with  Pay,  but  there  were  more  marked  differences  on  score  15  --  the 
Morale  measure.  Here,  the  three  Corps  locations  obtained  scores  of  38,  48,  and  53  percent  endorse¬ 
ment  of  favorable  response  alternatives.  While  these  differences  were  all  within  the  range  of  results 
obtained  at  other  organizations,  they  are  still  marked  differences  in  how  employees  describe  their  work 
environment. 

On  item  16,  Physical  Working  Conditions  and  Equipment,  all  three  Corps  locations  received  rela¬ 
tively  high  scores.  With  an  average  of  60  percent  favorable  responses,  one  of  the  Corps  locations  set  a 
new  high  over  those  scores  previously  obtained  in  any  of  the  other  18  organizations  studied.  A  some¬ 
what  similar  pattern  of  results  was  obtained  for  Co-worker  Cooperation  (item  17)  where  all  three  organ¬ 
izations  scored  relatively  high,  and  one  of  the  Corps  sites  tied  with  the  highest  score  in  any  of  the  other 
18  organizations  surveyed.  Since  Co-worker  Cooperation  is  a  key  determinant  of  Operational 
Efficiency,  this  score  represents  an  important  achievement  by  Corps  management. 

Another  important  result  demonstrating  the  effectiveness  of  Corps  management  was  obtained  on 
Operational  Efficiency,  where  all  three  Corps  sites  received  high  scores,  and  one  of  the  Corps  locations 
set  a  new  high  with  an  average  of  81  percent  of  the  favorable  response  alternatives  chosen.  This  clearly 
surpassed  the  previous  high,  where  an  average  of  75  percent  of  the  favorable  response  alternatives  had 
been  selected. 

The  final  score  from  the  MAS,  Workload  Balance,  indicated  some  level  of  strain  in  one  of  the 
three  Corps  locations,  where  only  29  percent  of  the  favorable  alternatives  (indicating  an  appropriate 
match  between  the  amount  of  work  and  the  number  of  employees)  were  chosen  This  result  was  a  new 
low  in  comparison  to  the  other  organizations  studied.  A  very  large  proportion  of  the  employees  at  this 
location  reported  that  they  were  understaffed.  The  other  two  locations  had  scores  of  39  and  46,  which 
were  in  the  range  obtained  for  the  other  18  organizations 
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A  review  of  the  results  indicated  nine  scores  where  the  overall  average  of  Corps  employees 
responding  favorably  was  below  50  percent.  This  is  the  actual  midpoint  and  helps  one  to  focus  on  score 
areas  which  employees  viewed  less  favorably.  Two  of  these  scores  were  in  areas  that  are  very  difficult 
for  Corps  management  to  change  Opportunities  for  Promotions  and  Satisfaction  With  Pay.  Of  course 
the  realities  and  constraints  that  affect  these  score  areas  need  to  be  fully  communicated  to  all  Corps 
employees. 

Responses  to  Individual  MAS  hems 

There  are  also  items  in  the  MAS  that  were  not  part  of  the  19  score  areas.  Among  these  were  the 
demographic  items,  specific  items  of  interest,  and  new  items  that  had  been  included  to  be  tested  for  use 
in  future  scales  of  the  MAS.  An  example  of  a  demographic  item  is  the  question  which  asks  whether 
the  employee  is  a  member  of  one  of  the  following  groups:  Black,  Spanish  or  Chicane,  American 
Indian,  Eskimo,  and  Aleut.  Seven  percent  of  the  employees  of  the  Corps  of  Engineers  sample  indi¬ 
cated  that  they  were  a  member  of  one  of  these  minorities. 

An  example  of  an  item  of  general  interest  is  one  which  asks,  "All  things  considered,  are  you 
satisfied  or  dissatisfied  with  the  work  you  do?"  Eighty-six  percent  of  the  Corps  employees  indicated  that 
they  were  satisfied  -  a  very  high  rating  of  satisfaction,  especially  considering  the  recent  literature  on  the 
subject.  Recent  studies  have  found  a  trend  toward  increasing  dissatisfaction  with  work  among  the 
nation’s  workers.22 

An  item  which  has  important  implications  for  the  future  of  the  Corps  asked  employees  to  describe 
the  effectiveness  of  the  career  development  program  in  the  Corps  of  Engineers:  14  percent  of  the 
employees  surveyed  responded  that  they  were  not  even  aware  of  the  program,  and  55  percent  indicated 
that  it  was  marginal  or  in  need  of  a  significant  amount  of  improvement.  The  remainder  (31  percent) 
described  the  career  development  program  as  effective  or  very  effective. 

Another  item  of  interest  is  the  one  which  asks,  "Of  all  the  supervisory  personnel  you  have  known 
in  the  Corps  of  Engineers,  what  percentage  have  been  effective  leaders?"  Twenty-three  percent  of  the 
employees  surveyed  replied  that  25  percent  or  fewer  supervisory  personnel  had  been  effective  leaders. 
Twenty-seven  percent  replied  that  between  26  and  50  percent  had  been  effective  leaders.  Thirty-three 
percent  replied  that  between  50  and  75  percent  of  the  supervisors  had  been  effective  leaders,  and  16 
percent  replied  that  over  75  percent  had  been  effective  leaders. 

About  one  half  the  employees  surveyed  indicated  that  fewer  than  50  percent  of  the  supervisory 
personnel  they  had  known  in  the  Corps  of  Engineers  were  effective  leaders.  These  results  offer  strong 
support  for  the  type  of  research  reported  here.  Improvement  of  Corps  management  requires  a 
thorough  understanding  of  the  strengths  and  weaknesses  of  current  management  practices,  a  commit¬ 
ment  to  excellence  on  the  part  of  all  echelons  of  the  organization,  and  planned,  realistic  strategies  for 
improvement. 

Follow-Up  Fisils 

Approximately  6  months  after  the  feedback  reports  had  been  distributed  to  the  supervisors  of  the 
work  groups  at  the  three  Corps  sites,  follow-up  visits  were  made  to 

1.  Determine  to  what  extent  the  MAS  had  been  used  as  a  management  tool  at  the  various  sites. 

2.  Determine  what  changes  need  to  be  made  in  the  MAS  system  for  it  to  most  benefit  the  Corps 
and  other  organizations. 

3.  Provide  a  gentle  reminder  to  supervisors  and  managers  that  the  Action  Plans  should  be  imple¬ 
mented  -  especially  since  a  readministration  of  the  MAS  was  already  scheduled  at  one  site 

The  follow-up  visits  lasted  2  days,  and  researchers  talked  to  as  many  people  as  could  be 
scheduled  The  interviews  were  with  different  levels  of  personnel  from  top-level  management  to 
employees.  The  key  findings  of  these  visits  were 

I.  Results  from  the  MAS  were  considered  valuable  enough  that  follow-up  administrations  are 
planned  at  one  site  and  already  scheduled  at  another. 

•”  M  R  Cooper.  B  S  Morgan.  P  M  tolcy.  and  (  B  Koplan.  “(  hanging  I  mploycc  Values  Deepening  Discontent  Hurnmt 
Business  Rivh'*'  (January  lcbruary  I'D*)!,  pp  IIT-I2S 


2.  If  strategies  for  implementing  Action  Plans  were  developed  more  systematically,  those  plans 
could  be  improved. 

3.  More  Executive  Office  support  of  the  MAS  is  needed,  and  that  support  should  be  evident  to 
employees.  For  example,  one  comment  was,  "If  we  had  known  at  the  beginning  that  the  survey  was 
this  important  and  that  there  would  be  follow-up  complete  with  visits  by  the  research  staff,  we  would 
have  made  a  greater  effort  to  get  the  data  in  and  do  something  with  the  feedback." 

4.  The  MAS  was  not  seen  as  just  another  survey.  As  a  result  of  this  administration,  supervisors 
made  many  individual  changes,  which  illustrated  the  intent  of  the  MAS  -  individual  and  organizational 
change  and  improvement. 

5.  Supervisory  personnel  became  more  aware  of  the  need  to  be  managers  rather  than  administra¬ 
tive  technicians. 

6.  If  the  MAS  had  been  administered  and  analyzed  by  units  within  the  Corps,  employees  would 
have  felt  less  free  to  express  their  opinions  about  managerial  practices. 

7.  The  facilitator  works  best  as  one  who  provides  information  instead  of  wielding  power. 

8.  To  do  an  effective  job,  one  facilitator  was  needed  for  every  50  work  groups. 


Future  Work  on  the  MAS 

While  the  MAS  system  has  many  benefits  and  definitely  can  be  useful  to  Corps  organizations, 
there  is  one  element  that  requires  some  modification  —  the  feedback  printout  received  by  individual 
supervisors.  In  their  present  form,  the  feedback  reports  are  extremely  difficult  for  some  supervisors  to 
understand.  Table  13  illustrates  the  same  information  found  on  the  existing  feedback  reports  but 
presents  it  in  a  format  which  is  much  clearer  and  easier  to  understand.  The  percentile  rank  scores  here 
indicate  a  direction  either  positive  (above  50,  shown  by  signs)  or  negative  (below  50,  shown  by  "-" 
signs),  and  the  score  areas  have  also  been  grouped  into  six  conceptual  categories  relevant  to  all  organi¬ 
zations.  The  organization's  percent  favorable  scores  are  still  included  on  the  far  right,  but  in  a  graphic 
form  rather  than  simple  numbers,  where  1  equals  20  percent  favorable,  2  equals  40  percent  favorable, 
and  so  on. 

For  example,  in  the  Management  score,  the  hypothetical  unit  reached  a  percentile  rank  of  68  for 
Supervisory  Effectiveness.  The  unit  also  had  approximately  50  percent  of  the  favorable  responses. 

Some  modification  of  the  score  areas  themselves  is  also  desirable.  The  communication  score  areas 
need  revision  to  reduce  the  overlap  of  the  actual  questions,  and  two  additional  score  areas  could  be 
added  -  Personnel  Development  and  Stress  Anxiety. 


Summary  and  Conclusions 

Although  the  three  sites  sampled  in  this  particular  study  may  not  be  totally  representative  of  all 
Corps  sites,  the  findings  presented  in  this  chapter  are  so  striking  that  they  command  attention. 

I  On  five  of  the  MAS  scores,  a  Corps  location  equalled  or  exceeded  the  highest  level  of  perfor¬ 
mance  obtained  at  any  of  the  18  other  organizations  studied  to  date  These  scores  were  Delegation  of 
Authority,  Equal  Opportunity  for  Minorities,  Physical  Working  Conditions  and  Equipment,  Co-worker 
Cooperation,  Operational  Efficiency. 

2.  On  three  of  the  MAS  scores,  a  Corps  location  scored  lower  than  the  minimum  obtained  at  any 
of  the  18  other  organizations.  These  scores  were  Planning  and  Administrative  Efficiency,  Performance 
Feedback,  Workload  Balance. 

3  On  one  MAS  score  -  Planning  and  Administrative  Efficiency  -  two  of  the  three  Corps  sites 
scored  lower  than  any  other  organization  ever  surveyed. 

4  On  another  MAS  score  -  Performance  Feedback  -  all  three  Corps  sites  obtained  the  lowest 
score  ever  obtained  on  any  MAS  administration 
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Table  13 


Revised  MAS  Feedback  Report 


MAS  RESULTS  FOR  t».S.  ARMY  CORPS  OF  ENGINEERS  JIU.  04.  1980  PAGE  123  A 

ORGANIZATION  CODES  WHICH  BEGIN  WITH 
EMPLOYEES  INCLUDED 


SCORE 

MANAGEMENT 

1.  SUPERVISORY  EFFECTIVENESS 

2.  PLANNING  &  ADMINISTRATIVE 

EFFICIENCY 

3.  TRAINING  EFFECTIVENESS 

4.  PERFORMANCE  FEEDBACK 
MAINTENANCE 

5.  FAIRNESS  OF  MANAGEMENT 

6.  WORK  SATISFACTION 

7.  MORALE 

8.  PERSONNEL  DEVELOPMENT  (NEW) 

9.  STRESS  AND  ANXIETY  (NEW) 
HYGIENE 

10.  OPPORTUNITY  FOR  PROMOTIONS 

11.  SATISFACTION  WITH  PAY 

2.  PHYSICAL  WORKING  CONDITIONS  & 
EQUIP. 

ADAPTATION 

13.  DELEGATION  OF  AUTHORITY 

14.  CLIMATE  FOR  INNOVATION 
PROGRAMS 

15.  EQUAL  OPPORTUNITY  FOR  WOMEN 

16.  EQUAL  OPPORTUNITY  FOR 

MINORITIES 

FUNDAMENTAL  SCORES 

17.  COMMUNICATIONS 

18.  CO-WORKER  COOPERATION 

19.  OPERATIONAL  EFFICIENCY 


PERCENTILE  RANK  OF  UNIT  SCORE  IN 

RELATION  TO  SCORES  OF  THE  TOTAL  GOAL  ATTAINMENT  OF 
SAMPLE  THE  TOTAL  SAMPLE 
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SPECIAL  NOTE  -  THIS  UNIT  IS  UNUSUALLY  UNDERSTAFFED.  THE  PERCENTAGE 
OF  RESPONDENTS  REPORTING  AN  EXCESSIVE  AMOUNT  OF  WORK  IN  RELATION  TO 
THE  NUMBER  OF  EMPLOYEES  WAS  87. 


The  statistical  analyses  performed  on  the  survey  instrument  and  the  data  obtained  through  its  use 
have  been  precise  and  exacting.  Due  to  their  complexity  and  high  level  of  sophistication,  specific- 
analytical  techniques  have  not  been  detailed  in  this  report,  however,  they  are  available  from  CERL-FS 
The  analyses  did  show  the  MAS  to  be  a  highly  reliable  and  valid  system  which  is  diagnostically  sensitive 
to  management  practices 
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8  SUMMARY 


Overview 

The  existing  performance  appraisal  system  and  the  selection  and  promotion  system  have  been 
investigated,  and  developmental  research  on  new  and  potentially  usable  procedures  for  both  systems 
has  been  completed.  These  new  procedures  have  been  described  in  this  report,  and  all  have  potential 
benefits  for  the  Corps  of  Engineers.  Each  procedure  involves  relatively  straightforward  activities  and 
has  demonstrated  validity  for  use  by  the  Corps. 


The  Selection  and  Promotion  Process 

In  Chapter  2,  the  conceptual  framework  for  all  research  described  in  this  report  was  presented  as 
Figure  1 .  Figure  8  shows  where  the  new  procedures  can  be  used  in  the  selection  and  promotion  pro¬ 
cess. 

Once  a  job  opening  is  identified,  the  job  requirements  are  defined  with  the  Job  Activities  Descrip¬ 
tion  before  candidates  for  the  position  are  screened.  When  the  actual  selection  or  screening  process 
begins,  two  sources  of  information  on  each  candidate  are  available  to  the  selection  official;  one  is  infor¬ 
mation  relevant  to  job  performance,  and  the  other  is  personal  information  for  each  candidate.  Job  per¬ 
formance  information  is  obtained  from  peer  rankings,  immediate  supervisor’s  ratings  (including  SK.AP 
elements),  and  second-level  supervisor’s  ratings.  The  personal  characteristics  are  described  by  the  Bio¬ 
graphical  Inventory,  the  Biographical  Sketch,  and  by  the  candidate  during  the  interview 


PERFORMANCE  ASSESSMENTS 
PH*  RANKINGS 
IMMEDIATE  SUPERVISOR 
RATINGS  -  SJCAP 
2nd  UVEl  SUPERVISOR 
RATINGS 


TRAINING  AND  DEVELOPMENT. 
PERSONNEL  MANAGEMENT  FOR 
EXECUTIVES 

MANAGEMENT  AUDIT  SURVEY 


PERSONAL  CHARACTERISTICS; 
biographical  inventory 
INTERVIEW 

biographical  sketch 


Figure  8.  Procedures  for  filling  an  open  position. 
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An  additional  procedure  --  which  is  not  a  selection  technique  per  se  --  is  the  Management  Audit 
Survey.  This  is  a  management  tool  which  may  be  used  to  measure  the  organization's  effectiveness. 


Review  of  the  Procedures 

The  Job  Activities  Description 

The  JAD  is  a  relatively  brief  instrument  which  provides  information  about  the  characteristics  of 
managerial  jobs.  The  relative  importance  of  various  job  duties  at  higher  levels  of  management  needs  to 
be  broadly  disseminated  so  that  information  on  these  characteristics  can  become  part  of  the 
selection/ promotion  process.  For  example,  ratings  showed  that  job  activities  such  as  training  and 
motivation  of  personnel  become  more  important  at  higher  levels  of  management.  Yet  interviews  indi¬ 
cated  that  managers  do  not  spend  a  significant  amount  of  time  on  employee  development  procedures. 

Besides  the  Training  and  Motivation  of  Personnel  category,  duties  in  the  Long-Range  Planning, 
Evaluation  and  Control  Activities,  and  Initiation  of  Improvements  of  Methods  and  Operations 
categories  were  significantly  related  to  managerial  level.  The  results  suggest  that  the  importance  of 
these  activities  should  be  emphasized  not  only  in  assessing  applicants  for  managerial  positions,  but  also 
in  helping  managers  develop  their  capabilities. 

Peer  Ranking 

This  process  asks  all  engineers  and  scientists  at  GS-12  and  above  to  rank  certain  scientific  and 
engineering  personnel  at  GS-U  and  above  whom  they  know  well  enough  on  two  criteria  dimensions 
Technical  Competence  and  Communication  Ability.  Such  a  system  seems  valuable  as  a  supplement  to 
present  procedures  for  appraising  performance.  Evaluations  of  individuals  by  other  managers  in  a 
variety  of  different  situations  should  be  included  in  the  selection  process.  Furthermore,  since  a  ranking 
procedure  would  be  involved,  peer  rankings  would  offset  the  leniency  of  some  ratings  by  supervisors, 
and  thus  would  help  identify  those  individuals  most  qualified  for  managerial  positions. 

Ratings  by  Immediate  Supervisors 

As  the  single  individual  most  familiar  with  a  subordinate’s  performance,  the  immediate  supervisor 
completes  a  form  which  rates  the  employee  on  each  of  the  five  criterion  dimensions  (including  the  two 
used  in  Peer  Ranking).  The  supervisor  is  also  asked  to  review  the  24  SK.AP  elements  and  indicate  the 
four  elements  on  which  the  employee  is  the  most  highly  skilled,  and  the  four  on  which  he/she  has  the 
greatest  opportunities  for  improvement.  This  procedure  forces  discrimination  on  the  part  of  the  super¬ 
visor,  thus  improving  diagnostic  sensitivity  and  making  the  assessment  more  useful  to  the  employee's 
career  development 

Second- Level  Supervisory  Ratings 

This  procedure  collects  ratings  from  second-level  supervisors  on  two  of  the  five  performance 
dimensions;  the  purpose  is  to  obtain  another  level  of  independent  assessments  of  performance.  The 
procedure  currently  used  with  the  SKAP  requires  that  second-level  supervisors  only  approve  the  ratings 
by  first-level  supervisors.  With  the  new  form,  the  second-level  supervisor  has  the  option  of  rating  the 
employee  on  up  to  an  1 1  -point  scale.  Each  even-numbered  point  has  a  behaviorally  based,  one- 
sentence  description  of  what  performance  at  that  level  should  indicate.  These  independent  ratings  give 
a  broader  assessment  base  and  additional  reliability  and  validity  to  the  entire  rating  process. 

The  Biographical  Inventory 

Form  V-2  of  the  Biographical  Inventory,  consisting  of  a  series  of  multiple-choice  questions,  pro¬ 
vides  information  from  employees  about  their  work  experience,  skills,  and  management  background  A 
variety  of  items  from  this  instrument  had  significant  relationships  with  a  composite  criterion  consisting 
of  a  promotion  rate  and  peer  nominations. 

The  biographical  items  associated  with  high  levels  of  managerial  performance  provide  a  partial  but 
important  portrait  of  the  successful  manager  in  the  Corps.  Successful  managers  are  comfortable  and 
effective  in  dealing  directly  with  people,  flexible  in  handling  a  variety  of  activities,  quick  witted,  and 
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able  to  handle  pressure  with  little  difficulty.  They  describe  themselves  as  skilled  in  managerial  activi¬ 
ties,  preferring  responsibility,  and  drawing  upon  the  breadth  of  experience  in  their  backgrounds.  They 
were  also  interested  and  effective  in  instituting  new  policies  and  procedures 

The  Biographical  Sketch 

The  Biographical  Sketch  is  a  one-page  instrument  designed  to  gather  specific  information  concern¬ 
ing  the  activities  of  and  rewards  received  by  the  individual  respondent.  The  12  items  contain  demo¬ 
graphic  information,  the  number  of  specific  types  of  awards  received,  and  the  activities  of  the  employee 
in  special  circumstances.  The  data  obtained  are  objective  and  can  be  verified  easily  from  personnel 
records. 

The  Interviewing  Manual  and  Guide 

Typically,  interviewing  procedures  tend  to  have  limited  validity  in  predicting  performance  criteria. 
Yet,  the  interview  is  one  of  the  most  widely  used  selection  procedures  because  both  applicants  and 
supervisors  need  to  communicate  their  expectations  and  exchange  information.  To  improve  interview¬ 
ing  skills,  a  guide  and  accompanying  instruction  manual  were  developed  during  this  research.  The 
guide  provides  questions  which  help  the  interviewer  gather  information  on  technical  job  requirements, 
managerial  characteristics  (including  data  related  to  the  Job  Activities  Description),  and  general  impres¬ 
sions.  The  guide  incorporates  some  sophisticated  procedures  for  recording  and  scoring  the  information 
gained  during  the  interview. 

The  Management  Audit  Survey 

The  MAS  is  a  system  to  stimulate  higher  levels  of  human  resource  management  by  providing  all 
levels  of  management  an  analysis  and  review  of  certain  organizational  characteristics.  The  MAS  system 
is  a  combination  of  a  100-item  questionnaire  completed  by  employees,  a  computerized  method  for  pro¬ 
viding  feedback  to  supervisors,  and  a  handbook  to  assist  supervisors  in  developing  programs  for  con¬ 
structive  change.  The  MAS  provides  quantified  indications  of  management  performance  and  organiza¬ 
tional  characteristics  for  areas  which  formerly  could  be  assessed  only  in  a  limited  or  subjective  manner. 
In  so  doing,  the  system  provides  the  impetus  for  a  review  of  current  practices  and  an  opportunity  to 
revitalize  the  organization.  The  MAS  is  not  to  be  directly  used  as  a  selection/ promotion  procedure,  but 
rather  is  a  management  tool  for  improving  the  total  organization  and  its  managers. 


Addressing  the  Problem  of  Performance  Criteria 

In  developing  any  performance  appraisal  system,  criteria  are  mandatory  to  assess  performance. 
Without  thorough  definition,  quantification,  analysis,  and  evaluation  of  performance  measures  (in  terms 
of  relevance,  reliability,  and  other  psychometric  characteristics)  any  selection/promotion  system  may 
have  crucial  weaknesses  in  practical  operation. 

Performance  criteria  serve  as  necessary  guides  in  establishing  a  system  for  monitoring  develop¬ 
ment,  evaluating  achievements,  diagnosing  deficiencies  or  opportunities  for  improvement,  providing 
feedback  abhut  process  or  outcome  measures,  and  integrating  these  concerns  for  decision  making  at  all 
levels.  In  the  studies  reported,  extensive  work  had  gone  into  developing  performance  criteria  which 
were  validated  against  the  performance  indicators  (peer  rankings,  immediate  supervisory  ratings, 
second-level  supervisory  ratings,  and  biographical  inventories).  The  new  procedures  are  diagnostically 
more  sensitive  and  have  significant  predictive  validity  because  of  the  criteria  development. 


Held  Evaluation  of  Managerial  Selection/Promotion  Procedures 

In  a  separate  contractual  effort,  the  procedures  described  and  discussed  in  this  report  were 
evaluated,  on  a  limited  basis,  at  two  Corps  sites.  Results  of  the  pilot  testing  are  available  in  a  report 
which  describes  that  work.23 


n  Fllison  R  l  ,  C  Abe.  D  G  Fox.  C  W  Guest,  and  W  I)  Veneklasen.  The  Development  and  Evaluation  of  Managerial 
Selection/Promotion  Procedures,  Technical  Report  P-1  IS  (CERl  .  W80) 
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9  CONCLUSIONS  AND  RECOMMENDATIONS 


This  report  has  documented  development  of  procedures  for  identifying  and  selecting  top-level 
managers  in  the  Corps  of  Engineers. 


The  Procedures  as  Individual  Components 

The  Job  Activities  Description 

This  procedure  is  straightforward  and  easily  used  to  determine  the  characteristics  of  managerial 
jobs.  It  has  gone  through  several  revisions  during  its  development,  with  results  demonstrating  the 
feasibility  of  using  the  JAD  across  grade  levels  and  job  classifications: 

1.  The  use  of  JADs  has  demonstrated  that  they  can  help  dn.: :rmine  the  job  skills  required  of  the 
person  chosen  to  fill  a  particular  position.  The  amount  of  time  spent  on  various  job  activities  was 
shown  to  vary  by  GS  level;  in  general,  higher  graded  personnel  spend  their  time  on  many  different 
managerial,  general  activities.  At  the  lower  grades,  however,  individuals  spent  most  of  their  time  per¬ 
forming  specialized  technical  duties. 

2.  The  importance  of  each  of  the  27  activities  was  also  presented  for  the  six  grade  levels  studied. 
At  grades  11  and  12,  no  management  activities  -  other  than  Use  Knowledge  and  Skills  in  a  Technical 
Capacity  -  were  rated  as  very  important.  At  grade  13,  eight  activities  were  rated  as  very  important,  by 
grade  16,  ten  activities  were  so  rated. 

3.  Beyond  the  general  findings  about  the  importance  of,  and  time  spent  on,  various  activities 
across  grade  levels,  the  analysis  of  job  classifications  was  not  conclusive. 

Peer  Ranking 

This  procedure  is  yet  untried  in  the  field,  but  its  rationale  is  sound. 

1.  Coworkers  are  likely  to  perceive  different  aspects  of  performance  than  are  supervisors 
Behavior  among  coworkers  is  likely  to  be  different  and  more  "natural"  than  behavior  in  the  presence  of 
a  supervisor. 

2.  Coworkers  are  in  a  better  position  to  observe  a  wider  variety  of  behaviors  because  they  usually 
have  more  contact  with  the  employee  than  does  the  supervisor. 

3.  Peer  ratings  can  be  used  as  a  reliability  check  on  the  performance  ratings  of  the  supervisor. 
First- Level  Supervisor's  Rating  Form 

This  instrument,  administered  in  the  field,  has  been  reviewed  by  Corps  supervisors  who  would  be 
using  it.  No  supervisor  indicated  that  he/she  would  have  any  problems  completing  the  form  on  any 
employee. 

The  form  is  brief  (a  single  sheet  of  paper)  and  easily  completed  by  the  supervisor.  Among  the 
advantages  of  this  procedure  are: 

1.  Each  performance  dimension  is  clearly  defined  so  that  all  employees  are  rated  on  the  same 
dimension;  each  is  independent  and  not  influenced  by  the  others. 

2.  The  rating  categories  within  each  dimension  are  discrete  and  compare  the  rated  individual  with 
all  other  employees  being  rated. 

3.  This  procedure  is  linked  to  the  existing  performance  evaluation  system  through  the  use  of  the 
24  SKAP  elements.  Four  areas  of  strengths  and  skills  and  four  areas  for  improvement  are  identified, 
allowing  the  supervisor  to  use  the  SKAP  elements  in  a  more  diagnostically  sensitive  manner. 

One  dimension  that  could  be  included  in  the  form  is  Supervisory  Ability  or  Potential  for  Super¬ 
visory  Status. 


Second- Level  Supervisor’s  Rating  Form 

This  device  was  also  reviewed  but  not  administered  in  the  field.  The  three  performance  dimen¬ 
sions  are  relevant  to  second-level  supervisors. 

While  more  elaborate  than  the  first-level  supervisor’s  rating  form,  this  instrument  does  have 
several  benefits: 

1.  All  employees  are  rated  at  the  same  time. 

2.  All  employees  are  rated  on  a  single  dimension  before  the  supervisor  proceeds  to  ratings  on  the 
next  dimension  This  technique  should  decrease  the  likelihood  that  the  second-level  supervisor  will 
rate  certain  employees  high  on  all  three  dimensions. 

3.  All  employees  are  rated  on  behaviorally  based  scales  rather  than  on  the  subjective  judgments 
of  the  supervisor. 

Ratings  and  Rankings  in  General 

Aside  from  the  individual  attributes  of  the  separate  rating  and  ranking  components,  the  instru¬ 
ments  provide  information  on  employees  which  has  never  been  systematically  available  to  Corps  super¬ 
visors. 

1.  The  performance  characteristics  or  dimensions  used  by  peers  and  the  two  supervisory  levels  in 
the  evaluation  process  are  the  same.  All  raters  and  rankers  use  the  same  definition  of  each  dimension. 

2.  By  obtaining  information  from  three  sources  on  the  individual  being  evaluated,  the  reliability 
of  the  ratings  is  much  improved  and  the  effects  of  overly  positive  or  overly  negative  single-source 
evaluations  are  eliminated. 

3.  Average  scores  per  rater  ate  now  systematically  provided. 

4.  The  diagnostic  sensitivity  of  the  existing  SK.AP  system  is  improved. 

5.  Second-level  supervisors  can  now  play  an  active  role  in  evaluating  employees  and  comparing 
them  rather  than  merely  signing  the  first-level  supervisor's  appraisal  form. 

Biographical  Inventory:  Form  V-2 

This  procedure  has  gone  through  at  least  two  major  revisions  in  its  development.  In  its  final 
form,  the  B1  provides  background  information  on  each  employee  in  six  conceptual  categories  relevant 
to  managerial  performance.  Use  of  biographical  inventories  has  several  advantages  for  the  Corps  of 
Engineers. 

1.  Previous  research  has  shown  this  approach  to  be  the  single  most  valid  predictor  of  future  per¬ 
formance. 

2.  Reliability  of  the  information  does  not  change  over  time. 

3.  Information  is  obtained  directly  from  the  individual  rather  than  interpreted  by  various  super¬ 
visory  levels. 

4.  The  objective  and  empirical  scoring  system  does  not  rely  on  value  judgments  of  the  informa¬ 
tion. 

Biographical  Sketch 

This  very  brief  form  compiles  information  on  each  employee's  activities  and  rewards.  Though  not 
a  substitute  for  any  existing  procedure,  the  Biographical  Sketch  has  the  following  benefits: 

1 .  Objective  information  is  easily  verified  by  personnel  records. 

2.  All  relevant  information  is  available  on  the  Sketch’s  single  page,  thus  eliminating  the  need  for 
a  time-consuming,  close  examination  of  the  entire  personnel  folder. 

3.  The  information  is  already  summarized  and  therefore  particularly  useful  to  selection  panels. 

To  improve  the  form,  two  additional  categories  could  be  included.  Suggestion  Awards  and  Profes¬ 
sional  Licensure. 


Interviewing  Manual  and  Guide 

Interviews  are  conducted  routinely  but  lack  reliability.  The  Interviewing  Guide  and  Manual  were 
designed  to  provide  a  number  of  improvements: 

1.  A  systematic  framework  for  obtaining  reliable  interview  information  from  each  candidate. 

2.  A  guide  which  can  help  bring  supervisors  to  a  shared  level  of  interviewing  skills. 

3.  A  scoring  technique  introducing  objectivity  and  standardization  to  interviews.  Each  candidate’s 
scores  can  then  be  considered  in  the  total  selection  process. 

Management  Audit  Survey 

This  procedure,  under  refinement  for  more  than  10  years,  is  designed  to  help  organizations 
achieve  higher  levels  of  effectiveness  and  efficiency.  It  is  not  a  selection  or  promotion  technique,  but 
can  be  used  to  upgrade  the  entire  organization: 

1  By  involving  all  levels  of  management  and  all  employees  in  identifying  and  overcoming  prob¬ 
lems  in  the  work  environment. 

2  By  stimulating  an  organization-wide  movement  toward  improved  management  practices, 
increased  productivity,  and  higher  levels  of  employee  satisfaction. 

3.  By  opening  communications  about  organizational  performance  and  effective  human  resource 
management 

The  MAS  was  shown  to  be  a  highly  reliable  and  valid  system  which  is  sensitive  to  management 
practices  and  their  impact  on  an  organization.  Use  of  the  MAS  is  a  low-risk  venture  that  could  be  quite 
valuable  to  the  Corps. 


Recommendations 

The  procedures  reported  here  promise  to  benefit  the  Corps  of  Engineers  in  varying  degrees 
Enough  developmental  and  feasibility  research  has  been  completed  to  recommend  that: 

1.  Additional  Divisions,  Districts  and  Field  Operating  Agencies  (FOAs)  be  surveyed  using  the 
MAS  This  will  help  assure  that  the  results  are  representative  of  the  Corps  as  a  whole. 

2.  The  Biographical  Inventory,  peer  ranking  system  and  supervisors'  appraisals  undergo  continued 
development  and  assessment  to  insure  their  validity  and  reliability. 

3.  These  tools  be  put  at  the  disposal  of  the  Corps  Division  and  District  Engineers  and  Chiefs  of 
FOAs  to  assist  them  in  meeting  their  managerial  improvement  objectives. 

4.  Interface  with  current  personnel  evaluation  systems  be  pursued  as  is  feasible  and  appropriate 

5.  Potential  opportunities  under  the  Civil  Service  Reform  Act  be  thoroughly  explored  to  include 
such  potentially  fruitful  areas  as  selecting  candidates  for  the  Senior  Executive  Service. 


65 


REFERENCES 


Campbell,  J.  P.,  M.  D.  Dunnette,  E.  E.  Lawler  Ill,  and  K.  E.  Weick,  Jr.,  Managerial  Behavior,  Perfor¬ 
mance,  and  Effectiveness  (McGraw-Hill,  1970).  pp  113-116. 

Carlson,  R.  E.,  "The  Current  Status  of  Judgmental  Techniques  in  Industry,"  in  W.  C.  Byham  and  D. 
Bobin,  eds.,  Alternatives  to  Paper  and  Pencil  Testing  (University  of  Pittsburgh,  Graduate  School  of 
Business,  1973). 

Cooper,  M.  R.,  B.  S.  Morgan,  P.  M.  Foley,  and  L.  B.  Koplan,  "Changing  Employee  Values.  Deepening 
Discontent?"  Harvard  Business  Review  (January-February  1979),  pp  117-125. 

Ellison,  R.  L.,  C.  Abe,  and  D.  G.  Fox,  The  Development  of  Preliminary  Performance  Indicators  for  the 
Selection  of  Managerial  Talent  in  the  U.S.  Army  Corps  of  Engineers,  Final  Report  DACA88-77-C- 
004  (U.S.  Army  Construction  Engineering  Research  Laboratory  [CERLl,  September  1977). 

Ellison,  R,  L.,  C.  Abe,  D.  G.  Fox,  and  K.  E.  Coray,  Validation  of  the  Management  Audit  Survey  Against 
Employment  Service  Criteria,  Final  Report  No.  DLMA  L-71-93  (U.S.  Department  of  Labor,  June 
1976)' 

Ellison,  R.  L.,  C.  Abe,  D.  G.  Fox,  C.  W.  Guest,  and  W.  D.  Veneklasen,  The  Development  and  Evalua¬ 
tion  of  Managerial  Selection/Promotion  Procedures,  Technical  Report  P-115  (CERL,  1980). 

Ellison,  R.  L.,  C.  Abe,  D.  G.  Fox,  and  W.  D.  Veneklasen,  The  Job  Activities  Description  Questionnaire: 
An  Analysis  of  Time  Spent  on  and  Importance  of  Managerial  Duties,  Interim  Report  E- 
157/ADA074175  (CERL,  September  1979). 

Ellison,  R.  L.,  L.  R.  James,  D.  G.  Fox,  and  C.  W.  Taylor,  The  Analysis  and  Prediction  of  Dow  Scientific 
Performance  (Final  report  submitted  to  Dow  Chemical  Co.,  1968). 

Ellison,  R.  L.,  L.  R.  James,  B.  W.  McDonald,  and  C.  W.  Taylor,  The  Prediction  of  Scientific  and 
Engineering  Performance  With  Biographical  Information  (Final  report  submitted  to  North  American 
Rockwell,  1968). 

Ellison,  R.  L.,  L.  R.  James,  T.  J.  Carron,  "Prediction  of  R&D  Performance  Criterion  With  Biographical 
Information,"  Journal  of  Industrial  Psychology,  Vol  5  (1970),  pp  37-57. 

Flanagan,  J.  C.,  "The  Critical  Incident  Technique,"  Psychological  Bulletin,  Vol  51  (1954),  pp  327-358. 

Ghiselli,  E.  E.,  Explorations  in  Managerial  Talent  (Pacific  Palisades  California:  Goodyear  Publishing, 
1971). 

Hemphill,  J.  K.,  ed..  The  Engineering  Study  (Educational  Testing  Service,  1963). 

Hemphill,  J.  K.,  "Job  Descriptions  for  Executives,"  Harvard  Business  Review,  Vol  37  (1959),  pp  55-67. 

Hollander,  E.  P.,  "Validity  of  Peer  Nominations  in  Predicting  a  Distinct  Performance  Criterion,"  Journal 
of  Applied  Psychology,  Vol  49  (1965),  pp  434-438. 

Klimoski,  R.  J.  and  M.  London,  "The  Role  of  the  Rater  in  Performance  Appraisal,"  Journal  of  Applied 
Psychology,  Vol  59  (1974),  pp  445-451. 


66 


Kraut,  A.  1.,  "Prediction  of  Managerial  Success  by  Peer  and  Training-Staff  Ratings,”  Journal  of  Applied 
Psychology ,  Vol  60  (1975),  pp  14-19. 

Laurent,  H.,  "Cross-Cultural  Cross-Validation  of  Empirically  Validated  Tests,"  Journal  of  Applied 
Psychology ,  Vol  54  (1970),  pp  417-423. 

Laurent,  H.,  "Early  Identification  of  Managers."  Management  Record,  Vol  24  (1962),  pp  33-38. 

Laurent,  H.,  "Research  on  the  Identification  of  Management  Potential,"  in  J.  A.  Myers,  Jr.,  ed.. 
Predicting  Managerial  Success  (Ann  Arbor,  M;  Foundation  for  Research  on  Human  Behavior, 
1968). 

Lawler,  E.  E.,  Ill,  "Management  Performance  as  Seen  from  Above,  Below,  and  Within,”  Journal  of 
Applied  Psychology,  Vol  51  (1967),  pp  247-253. 

Mahoney,  T.  A.,  Criteria  of  Organizational  Effectiveness,  Mimeographed  report  (University  of  Minnesota, 
Industrial  Relations  Center,  1966). 

Miner,  J.  B.  and  M.  G.  Miner,  Personnel  and  Industrial  Relations:  A  Managerial  Approach  (MacMillian, 
1977). 

Oberg,  W.,  "Making  Performance  Appraisal  Relevant,"  Harvard  Business  Review,  Vol  50,  No.  1  (1972), 
pp  61-67. 

Weitz,  J.,  "Selecting  Supervisors  with  Peer  Ratings,"  Personnel  Psychology,  Vol  11  (1958),  pp  25-36. 


67 


APPENDIX  A: 


IMMEDIATE  SUPERVISOR’S  ASSESSMENT  FORM 

IMMEDIATE  SUPERVISORY  ASSESSMENT 
(For  Research  Purposes  Only) 


Introduction 

As  part  of  a  study  on  the  development  of  performance  indicators  for  managerial  talent,  a  mean¬ 
ingful  evaluation  of  all  scientists  and  engineers  (GS-11  and  higher)  by  their  immediate  supervisor  is 
necessary.  Such  information  will  play  a  vital  role  in  evaluating  other  data  being  collected.  TTiis  can  be 
accomplished  by  completing  both  sides  of  the  Supervisory  Assessment  form.  The  information  you  pro¬ 
vide  will  be  used  in  the  development  of  a  system  to  help  the  Corps  of  Engineers  achieve  a  cadre  of 
strong  leadership  in  the  years  to  come.  The  data  will  be  used  for  research  purposes  only,  and  will  be 
processed  by  the  Institute  for  Behavioral  Research  in  Creativity  (IBRIC). 


Instructions 

Please  fill  out  a  separate  Supervisory  Assessment  form  for  each  employee  you  directly  supervise. 
Please  write  the  name  of  the  employee  and  your  name  at  the  top  of  each  form  in  the  areas  indicated. 

On  side  1,  rate  the  employee  on  each  of  five  dimensions,  using  the  scale  at  the  right.  The  scale 
indicates  that,  of  all  the  employees  ihat  you  have  known  in  the  Corps,  only  the  top  10%  would  be  rated 
as  outstanding,  the  next  25%  would  be  rated  as  unusually  competent,  the  middle  30%  as  fully  com¬ 
petent,  etc.  Your  ratings  should  use  the  scale  to  reflect  your  observations  about  differences  in 
employee  performance. 

The  rating  that  you  assign  to  an  employee  on  one  dimension  should  not  influence  your  rating  of 
that  employee  on  any  other  dimension  -  each  employee  probably  has  areas  of  strength  and  areas  of 
weakness.  Place  a  check  mark  in  the  box  that  corresponds  to  your  rating  of  the  individual  on  the 
dimension. 

On  side  2  of  the  form,  you  are  to  indicate  for  the  individual  the  four  (4)  areas  in  which  he/she  is 
the  most  highly  skilled,  and  the  four  (4)  areas  in  which  the  individual  needs  improvement  the  most 
Write  the  appropriate  response  on  the  line  to  the  left  of  the  number  for  the  characteristics  identified, 
following  the  directions  on  the  top  of  side  2. 

Your  cooperation  is  greatly  appreciated. 


SUPERVISORY  ASSESSMENT 
(For  Research  Purposes  Only) 

Name  o  f 
Employee 


Immediate 

Supervisor 


_ JOB  DIMENSIONS _  _ _ ASSESSMENT 

For  each  job  dimension,  place  a  check  mark  in  the  box  under 
the  ONE  category  which  most  accurately  describes  the 
employee's  ability  in  that  area. 

DO  NOT  let  your  rating  on  one  dimension  influence  your 
rating  on  other  dimensions. 


1.  TECHNICAL  COMPETENCE.  Consider  the  employee's 
ability  to  apply  comprehensive  knowledge  to  the 
solution  of  complex  engineering  or  scientific 
problems.  Evaluate  the  employee's  methodological 
expertise  in  developing  effective  and  economical 
use  of  materials  and  forces  of  nature,  and 
competence  in  scientific  and  engineering  functions. 


2. 


COMMUNICATIONS.  Evaluate  the  employee’s  ability 
to  receive,  comprehend,  and  transmit  a  range  of 
messages  to  a  variety  of  audiences.  Include  ability 
to  listen  and  sense  the  needs  of  others,  establish 
rapport,  and  use  effective  techniques  to  see  that 
he/s he  understands  and  is  understood.  Consider  the 
power  the  individual's  communications  have  to 
influence  audiences,  group  or  individual. 


i 
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3. 


COORDINATION.  Consider  how  well  the  employee 
analyzes  work  flow  and  recognizes  interaction  of 
assigned  work  with  actions  of  others;  the  degree 
to  which  the  employee  fully  understands  the  necessity 
for  team  effort  in  problem  solving,  and  accomplishes 
work  by  actively  mobilizing  staff  capability  and 
obtaining  needed  assistance  from  all  echelons. 


4  . 


DECISION  MAKING.  Evaluate  the  person's  intellectual 
skills  in  decision  making.  Consider  the  degree  to 
which  the  person  gathers  meaningful  information, 
weighs  it  according  to  relevance  and  reliability, 
and  is  analytical  in  the  development  and  application 
of  appropriate  criterion  factors  so  that  highly 
effective  decisions  on  complex  issues  result. 


5. 


INNOVATION.  Assess  whether  the  individual  typically 
approaches  situations  with  an  openness  to  fresh 
ideas  which  facilitate  innovations,  organizational 
vitality,  and  responsiveness  to  change.  Consider 
how  well  informed  the  individual  is  of  current 
developments  within  and  outside  the  organization 
and  their  implications  for  future  organizational 
development . 
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On  the  following  list  of  employee  characteristics,  please  mark  with  an  "S"  those 
four  (4)  on  which  the  employee  is  the  most  highly  skilled.  Also,  please  mark  with 
an  ”1"  those  four  (4)  on  which  the  employee  has  the  greatest  opportunity  for  improve¬ 
ment.  Please  remember  to  mark  four  and  only  four  with  each  letter.  No  single  char¬ 
acteristic  should  be  marked  with  both  an  "S"  and  an  “I".  The  list  of  characteristics 
was  taken  from  the  Annual  Referral  Assessment  —  SKAP  (DA  Form  4428-R) .  If  you  would 
like  a  more  specific  definition  of  the  characteristics,  please  refer  to  that  form. 


1.  Skill  in  establishing  and 
maintaining  effective 
working  relationships 

2.  Skill  in  oral  communica¬ 
tion  and  briefings 

3.  Skill  in  written  communi¬ 
cations 

4.  Dependability 

5.  Initiative  and  drive 

6.  Ability  to  absorb  new  facts 
and  concepts 

7.  Knowledge  of  automatic 
data  processing 

8.  Motivated  to  self-improve¬ 
ment 

9.  Personal  commitment  to  Equal 
Employment  Opportunity,  in¬ 
centive  awards,  career 
development,  and  special 
emphasis  programs 

10.  Ability  to  select,  develop, 
motivate,  and  supervise 
subordinates  of  varied 
backgrounds  and  skills 

11.  Ability  to  delegate  authority 

12.  Ability  to  coordinate  and 
integrate  work  programs  of 
subordinate  organizations, 
peer  groups,  and  higher 
authority 

13.  Ability  to  assess  program 
requirements  and  to  adjust 
available  resources  for  opti¬ 
mum  efficiency  and  effective¬ 
ness 


14.  Ability  to  relate  organization 
and  mission  requirements  to 
training  needs 

15.  Familiarity  with  personnel  poli¬ 
cies  and  procedures 

16.  Technical  competence  in  assigned 
activities 

17.  Knowledge  and  technical  appli¬ 
cation  of  applicable  laws, 
policies,  regulations,  and 
procedures 

18.  Knowledge  of  contract  law  and 
contract  provisions  related  to 
assigned  activities 

19.  Ability  to  execute  technical 
activities  within  established 
financial  and/or  time  constraints 

20.  Ability  to  render  sound  decisions 
in  a  timely  manner  on  technical 
matters 

21.  Ability  to  accomplish  program¬ 
ming  and  budgeting  activities, 
including  long-range  planning 

22.  Ability  to  understand  the  inter¬ 
relationship  of  socio-economic 
and  environmental  goals  and 
implement  coordination  required 
in  the  use  of  water  and  land 
resources 

23.  Familiarity  with  functions  of 
industry  and  others  related  to 
assigned  activities 

24.  Ability  to  apply  methods  and 
procedures  necessary  to  assure 
quality  end  product 
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APPENDIX  B: 


SECOND-LEVEL  SUPERVISOR’S  RATING  FORM 


RATING  FORMS  FOR  CORPS  OF  ENGINEERS  EMPLOYEES 
(For  Research  Purposes  Only) 


Introduction 

This  rating  form  is  an  important  part  of  a  study  on  the  development  of  performance  indicators  for 
managerial  talent.  More  specifically,  sources  of  information,  predictor  tools,  and  methods  of  quantify¬ 
ing  performance  information  will  be  evaluated  as  part  of  the  study  to  determine  how  improved 
selection/ promotion  procedures  for  managerial  positions  can  be  developed. 

All  information  collected  during  the  study  will  be  kept  strictly  confidential  and  will  be  used  for 
research  purposes  only.  The  data  will  be  processed  entirely  at  the  Institute  for  Behavioral  Research  in 
Creativity  (IBRIC),  and  the  results  will  be  reported  only  in  terms  of  relationships  found  on  the  total 
group,  without  any  reference  whatever  to  individual  employees. 


Explanation  of  the  Rating  Forms 

As  you  open  the  accompanying  booklet,  you  will  find  a  space  provided  on  the  left  for  the  names 
of  all  individuals  for  whom  you  serve  as  a  second  level  supervisor.  On  the  right  of  the  page,  five 
different  levels  of  a  job  performance  dimension  are  described.  These  descriptions  occur  at  even  points 
along  an  11 -point  scale.  The  odd  numbers  between  the  statements  indicate  a  position  on  the  scale 
between  the  two  adjacent  statements.  These  odd  numbers  can  be  used  to  indicate  on  which  side  of  a 
particular  statement  an  employee’s  performance  tends  to  be.  They  also  can  be  used  to  make  distinc¬ 
tions  between  any  two  employees  who  receive  the  same  rating  so  that  you  can  reduce  the  number  of 
ties  in  your  ratings  on  each  page. 

The  percentages  in  the  vertical  block  present  a  picture  of  how  100  employees  would  be  distributed 
across  the  descriptive  levels.  These  percentages  are  included  as  guidelines  as  you  make  your  evalua¬ 
tions. 

Please  carefully  read  the  instructions  on  the  cover  page  before  beginning  the  rating  procedure. 


RATING  FORMS  FOR  CORPS  OF  ENGINEERS  EMPLOYEES 
(For  Research  Purposes  Only) 


Instructions  for  Completing  the  Forms 


The  following  four  steps  are  required  to  complete  the  rating  forms: 

1.  On  each  rating  scale,  read  all  of  the  five  descriptive  statements  thoroughly  before  rating  the 
first  employee.  Select  the  number  from  the  scale  which  best  describes  the  performance  of  each 
employee  and  write  this  number  within  the  parentheses  opposite  that  name.  Rate  all  of  the  people  on 
each  scale  before  proceeding  to  the  next  page.  You  may  rightly  think  that  none  of  the  statements  is  an 
exact  description  of  the  individual,  but  make  the  best  single  choice  you  can. 

2.  In  your  assessments,  evaluate  each  person  in  comparison  to  the  descriptive  statements  and  the 
percentage  guidelines,  emphasizing  demonstrated  accomplishments  rather  than  potential.  If  you  have 
not  had  a  sufficient  opportunity  to  observe  an  individual’s  performance,  write  NO  (no  opportunity)  in 
the  parentheses. 

3.  In  your  evaluations,  be  certain  to  remember  that: 

Everyone  has  both  strengths  and  weaknesses;  therefore 
the  usual  pattern  for  an  individual  should  reflect  a 
combination  of  different  ratings.  It  would  be  a  rare 
instance  when  an  employee  would  be  equally  strong  or 
weak  in  both  aspects  of  performance. 

Ties  should  be  reduced  by  using  the  odd  number  between 
two  adjacent  statements. 

4.  After  you  have  completed  the  rating  forms,  remove  the  back  sheet  which  lists  the  names  of 
those  people  you  have  rated.  This  will  help  to  ensure  the  anonymity  of  all  participants. 

Remember,  the  ratings  are  for  research  purposes  only  and  results  will  be  reported  in  terms  of 
relationships  found  on  the  total  group,  without  any  reference  whatever  to  individuals,  so  please  make 
your  evaluations  as  accurate  as  possible. 

Thank  you  for  your  cooperation. 

Rater  Number _ 


Write  the  number  from  the 
scale  which  best  describes 
each  employee  in  the 
parentheses  beside  his/her 
name. 

Names  of  persons  to  be  rated: 


DECISION  MAKING 

Evaluate  the  person's  intellectual  skills  in 
decision  makinq.  Consider  the  degree  to  which 
the  person  gathers  meaningful  information  and  is 
analytical  in  the  development  and  application  of 
appropriate  criterion  factors  so  that  highly 
effective  decisions  on  complex  issues  result. 

DO  NOT  consider  any  other  aspects  of  performance-- 
ONLY  the  person's  decision-making  performance. 


(  ) 

1 

(  ) 

(  )  M  Has  some  difficulty  in  integrating  informa- 

2  2  tion,  in  considering  significant  factors, 

and  in  making  important,  timely  decisions. 
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Available  information  and  criteria  for 
A  problem  solving  are  handled  effectively  for 
most  routine  Droblems  and  decisions. 


Collects  meaningful  data  and  weicns  'nfor- 
g  mation  appropriately  so  that  most  decisions 
are  handled  satisfactorily;  judgements  are 
usually  correct. 
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Uses  information  appropriatel  v  and  ras 
superior  abilitv  to  analyze  and  -ecoqni;e 
implications  which  leads  to  elective 
deci sions . 


O  -C 


(  ) 
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„  Evaluates  information  and  jeve'ODS  ■'•levant 

2  criteria  so  that  even  complex  decisions  ar? 

precedent  settino.  Makes  unusual!’/  valid 
evaluations. 


( 
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INNOVATION 


Write  the  number  from  the 
scale  which  best  describes 
each  employee  in  the 
parentheses  beside  his/her 
name. 

Names  of  persons  to  be  rated: 


Evaluate  the  innovative  impact  of  the  person's 
efforts.  Consider  the  extent  to  which  situations 
are  approached  with  an  openness  to  new  ideas 
which  facilitates  innovations,  organizational 
vitality,  and  responsiveness  to  change.  DO  NOT 
consider  any  other  aspects  of  the  person's  job 
performance— ONLY  the  innovative  effects  of  the 
individual's  job  performance. 


1 


2  10 


In  discussions  about  new  approaches,  tends  to 
emphasize  difficulties  rather  than  oDportunities 
does  not  really  want  or  support  innovative  ideas 


Occasionally  supports  some  new  procedures,  but 
usually  urges  a  careful  evaluative  approach; 
relies  upon  current  methodology  rather  than 
promoting  or  developing  new  ideas. 


Periodically  develops,  or  provides  some 
support  for,  new  ideas,  but  is  basically 
oriented  toward  current  procedures  pending 
evaluative  evidence. 


Carefully  reviews  ongoing  and  new  procedures, 
looking  for  possible  improvements.  Often 
auestions  status  auo  to  achieve  organizational 
vitality. 


Usually  is  enthusiastic,  open,  and  supportive 
of  new  procedures;  works  actively  to  promote 
new  responses  and  aporoaches  which  facilitate 
organizational  effectiveness. 


74 


APPENDIX  C: 


SAMPLE  INSTRUCTIONS  FOR  RANKING  PROCESS 


RANKING  SCALES  FOR  CORPS  OF  ENGINEERS 
(For  Research  Purposes  Only) 

The  attached  forms  are  part  of  a  study  to  try  to  improve  the  selection/ promotion  process.  These 
forms  were  designed  to  determine  whether  rankings  of  individuals  by  their  peers  would  provide  useful 
information  to  an  overall  system  of  performance  evaluation  and  career  development.  The  process  is 
concerned  with  assessing  individuals  at  GS-1 1  and  above.  The  current  activity  is  for  research  purposes 
only,  these  current  evaluations  will  be  kept  strictly  confidential.  The  data  will  be  processed  entirely  at 
the  Institute  for  Behavioral  Research  in  Creativity,  and  the  results  will  be  reported  only  in  terms  of 
total  group  relationships.  NO  results  will  be  reported  on  individuals. 

Studies  in  other  settings  indicate  the  peer  ranking  process  has  the  potential  of  providing  important 
information  and  this  information  is  from  individuals  other  than  the  immediate  supervisor.  If,  after  a 
review  of  the  current  research  results,  it  is  concluded  that  rankings  by  peers  do  provide  useful  informa¬ 
tion,  then  the  peer  ranking  procedure  would  become  part  of  the  selection  system  along  with  the  SKAP 
procedures,  panelling  procedures,  etc. 

Please  work  by  yourself.  The  ranking  process  must  be  done  by  each  person  independently.  Since 
judges  often  differ  in  their  evaluations  of  the  same  individual,  the  rank  orders  are  not  expected  to  be 
the  same  for  all  judges.  Moreover,  since  communication  skills  and  technical  competence  represent 
different  abilities,  it  is  expected  that  the  two  rank  orders  will  usually  be  different. 

To  facilitate  the  ranking  process,  you  have  been  provided  with  a  deck  of  60  computer  cards.  Each 
card  has  the  name  of  one  individual,  chosen  randomly  from  all  GS-1  Is  and  above. 


Step-by-Step  Instructions 


1 .  Take  the  deck  of  computer  cards  and  rapidly  sort  them  into  the  two  groups  described  below: 

a)  Those  whose  performance  you  know  well  enough  to  evaluate  on  technical  competence 
and  communication  skills.  These  cards  should  represent  persons  whom  you  have  actu¬ 
ally  observed  at  work.  These  are  the  ones  you  will  rank. 

b)  Those  whose  performance  you  do  not  know  well  enough  to  evaluate.  This  group  should 
also  include  yourself  and  any  of  your  subordinates,  if  they  appear  in  the  deck.  These 
persons  will  not  be  ranked;  all  the  cards  in  this  group  should  be  thrown  away.  In  most 
cases  this  "discard"  group  will  be  a  substantial  portion  of  the  original  60  cards. 

2.  Carefully  read  the  definition  of  "technical  competence." 

3.  Take  the  deck  of  cards  representing  the  persons  whose  work  is  well-known  to  you.  Use  the  cards 
to  rank  order  these  people  from  highest  to  lowest  on  "technical  competence."  One  way  to  do  this 
would  be  to  take  the  top  card  and  place  it  face  up  on  the  table,  take  the  second  card  and  compare 
it  to  the  first  card,  placing  it  to  the  left  of  the  first  card  if  the  person  has  more  technical  com- 
tence  or  to  the  right  if  the  person  has  less.  Continue  this  process,  opening  up  space  between 

the  cards  whenever  necessary,  until  all  the  cards  are  ranked  in  order.  Now  check  the  rankings  and 
make  any  final  changes  in  rank  order.  Assemble  the  deck  so  that  the  highest  rank  is  on  top  -  the 
person  with  the  most  technical  competence.  The  lowest  rank  should  be  on  the  bottom. 


4.  On  the  left-  hand  side  of  each  card,  record  the  number  representing  that  person’s  rank  on  technical 
competence.  The  top-ranked  person  should  be  T;  if  you  rank  12  persons,  the  bottom-ranked  per¬ 
son  should  be  "12”.  Write  each  number  so  that  it  is  large  and  clear. 

5.  Check  to  ensure  that  the  bottom  rank  is  given  the  same  number  as  the  number  of  persons  you 
ranked. 

6.  Shuffle  the  deck. 

7.  Carefully  read  the  definition  of  "communications." 

8.  Again,  take  the  deck  of  cards  representing  the  persons  whose  work  is  well-known  to  you.  Use  the 
cards  to  rank  order  these  people  from  highest  to  lowest  on  "communications."  Assemble  the  deck 
so  that  the  highest  rank  is  on  top  -  the  person  with  the  best  communication  skills.  The  lowest 
rank  should  be  on  the  bottom.  Check  the  rankings  and  make  any  final  changes  in  rank  order. 

9.  On  the  right-hand  side  of  each  card,  record  the  number  representing  that  person’s  rank  on  com¬ 
munications.  Again,  the  top-ranked  person  should  be  "1".  Write  each  number  so  that  it  is  large 
and  clear. 

10.  Check  to  insure  that  the  bottom  rank  given  is  the  same  as  the  number  of  persons  you  have  ranked. 

11.  Check  to  insure  that  each  card  has  two  numbers  written  on  it  -  the  rank  order  for  "technical  com¬ 
petence"  on  the  left,  the  rank  order  for  "communications”  on  the  right. 

12.  Return  your  ranked  deck  of  cards  to .  . . ■ 


APPENDIX  D: 


BIOGRAPHICAL  INVENTORY:  FORM  V 

This  preliminary  examination  of  a  BJ  as  an  integral  part  of  the  process  of  selecting  high-level 
managers  is  a  study  within  itself.  As  mentioned  in  Chapter  5,  the  use  of  biographical  data  has  been  one 
of  the  most  valid  approaches  to  identifying  managerial  talent.  Data  for  this  study  were  collected  on  2 1 9 
scientific  and  engineering  employees  at  GS  levels  11  through  16  from  the  Missouri  River  Division  and 
the  Omaha  District. 


Description  of  the  Biographical  Items 

Of  the  150  items  in  Form  V,  54  were  obtained  from  forms  the  contractor  previously  developed 
for  other  studies.  Thirty-three  of  these  were  taken  from  a  B1  developed  for  a  study  of  performance  cri¬ 
teria  -  for  example,  creativity,  skill  with  people  -  among  scientists  and  engineers  at  a  large  aerospace 
firm.  Twenty  items  were  taken  from  a  form  developed  to  assess  performance  of  first-level  supervisors 
at  a  large  textile  manufacturing  and  development  company. 

Fifty  biographical  items  were  developed  specifically  for  this  current  study.  These  new  items  were 
written  to  supplement  those  from  other  Bis  and  to  cover  more  adequately  ideas  obtained  from  the 
interviews  with  Corps  of  Engineers  employees. 

In  addition,  10  items  in  Form  V  were  revised  versions  of  items  taken  from  Ghiselli's  Self 
Description  Inventory.24  These  items  ask  the  respondent  to  indicate  which  of  two  traits  is  more 
descriptive  of  him/herself.  These  items  were  empirically  keyed  by  Ghiselli  to  predict  various  aspects  of 
managerial  performance,  such  as  supervisory  ability,  intelligence,  initiative,  self-assurance,  and 
decisiveness. 

In  Form  V,  nine  items  were  revisions  of  items  taken  from  the  Standard  Oil  of  New  Jersey. 
Management  Judgment  Test.25  The  Management  Judgment  Test  has  demonstrated  significant  validities 
in  predicting  managerial  performance.  For  example,  correlations  of  .47  and  .51  were  obtained  on  two 
separate  samples  with  the  Management  Judgment  Test  for  predicting  an  overall  success  index.  These 
validities  were  exceeded  only  by  those  for  biographical  data.  The  overall  success  index  was  based  on 
the  manager’s  position  level,  his/her  salary  history,  and  an  effectiveness  ranking  by  higher  level  people 
-  with  precautions  to  eliminate  the  effects  of  his/her  age,  length  of  experience,  etc.  An  important 
component  of  this  measure  was  the  manager's  past  flexibility  and  innovation  in  adapting  to  changing 
conditions. 

Twenty-seven  items  in  Form  V  were  concerned  with  breadth  of  experience.  These  items  asked 
about  the  amount  of  experience  the  respondent  had  in  each  of  seven  Corps  functional  divisions  and 
twenty  functional  areas.  Lists  of  these  experiential  categories  were  obtained  from  Department  of  the 
Army  forms. 

Throughout  the  construction  of  Form  V,  the  guidelines  for  selection  devices  put  forth  by  the  Civil 
Service  Commission  (now  Office  of  Personnel  Management)  were  strictly  followed.  This  meant  that 
items  had  to  have  either  a  relationship  with  actual  managerial  performance  or  demonstrated  statistical 
validity  in  the  past. 

The  items  in  Form  V  could  be  subjectively  assigned  to  10  categories,  plus  a  miscellaneous 
category: 

1.  Achievement  Motivation.  Eleven  items  dealt  with  the  willingness  of  the  respondent  to  make 
the  effort  necessary  to  get  ahead.  Questions  concerned  the  energy  with  which  the  respondent  conducts 
his/her  duties,  the  desire  to  master  a  broad  body  of  knowledge,  and  how  favorably  the  employee  felt 
he/she  would  compare  to  others  as  an  executive. 


!>  F.  E  Ghiselli.  Explorations  in  Managerial  Talent  (Pacific  Palisades  California:  Goodyear  Publishing.  1971). 

!'J  P  Campbell,  M  D.  Dunnetle,  E.  E.  Lawler  111,  and  K.  E  Wetck,  Jr,,  Managerial  Behavior,  Performance,  and  Eflecnvrness 
(McGraw-Hill.  1970) 
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2.  Antecedent  Leadership.  Four  items  were  concerned  with  the  respondent’s  having  been 
elected  to  positions  of  leadership  while  in  college. 

3.  Communications.  Twelve  items  in  the  form  concerned  the  respondent’s  ability  to  communi¬ 
cate  with  others.  Questions  were  asked  about  the  respondent’s  history  of  speaking  before  groups,  writ¬ 
ing  skills  and  experience,  and  style  of  communication. 

4.  Intelligence.  Fourteen  items  were  related  to  the  respondent’s  ability  to  grasp  ideas  and  use 
them  in  rational  thinking,  to  relate  to  several  different  activities  at  the  same  time,  to  ask  meaningful 
questions,  to  be  "quick  wilted." 

5.  Initiative.  Fifteen  items  were  concerned  with  the  development  and  implementation  of  new 
methods,  processes,  and  ideas.  Specific  questions  asked  about  the  persistence  and  aggressiveness  of  the 
respondent  in  standing  up  for  and  gaining  recognition  for  ideas,  ability  to  set  a  course  and  move  toward 
a  goal  with  little  guidance,  and  willingness  to  work  within  the  system  or  work  to  change  the  rules  if 
necessary  to  implement  meaningful  change. 

6.  Interpersonal  Skills.  Thirteen  items  dealt  with  interacting  with  others,  gaining  cooperation 
and  respect,  being  tactful,  getting  to  know  people,  and,  in  general,  being  interested  in  others. 

7.  Management  Ability.  Twelve  items  dealt  with  management  ability,  such  as  the  skill  to  medi¬ 
ate  a  dispute,  stimulate  productive  thought  and  action,  anticipate  potential  problems,  forecast  program 
development. 

8.  Management  Judgment  Fifteen  items  were  concerned  with  presenting  a  number  of  alterna¬ 
tives  for  handling  a  specific  problem;  the  respondent  was  asked  to  judge  which  alternative  would  pro¬ 
duce  the  best  results. 

9.  Job  Activities.  Seventeen  items  were  concerned  with  points  taken  specifically  from  the  Job 
Activities  Description  form.  These  items  presented  pairs  of  activities,  one  of  which  was  more  typically 
associated  with  higher-level  managers,  and  the  respondent  was  asked  to  indicate  in  which  of  the  two 
he/she  was  more  skilled. 

10.  Experience.  Thirty  items  dealt  with  the  respondents’  experience  in  the  Corps.  Of  these 
items,  seven  concerned  specific  Corps  of  Engineers  Division  functions;  20  concerned  the  various  func¬ 
tional  areas  listed  in  the  Department  of  the  Army  Form  4428.  These  27  items  asked  the  amount  of 
time  spent  in  each  of  the  areas.  Additional  items  asked  in  how  many  of  the  functional  divisions  they 
had  experience,  in  how  many  of  the  functional  areas  they  had  experience,  and  the  amount  of  foreign 
country  or  overseas  experience  they  had. 

11.  Miscellaneous.  Seven  items  could  not  be  assigned  to  any  of  the  above  categories.  These 
asked  such  things  as  how  the  respondents  felt  about  filling  out  the  questionnaire;  how  much  education 
they  had  completed;  how  their  time  was  distributed  across  technical  and  managerial  duties,  how  they 
felt  about  jumping  from  one  job  activity  to  another. 


Other  Measures  Obtained  for  the  BI  Study 

In  addition  to  the  responses  to  the  1 50  BI  Form  V  items,  those  people  who  completed  the  Bl  were 
asked  to  provide  information  about  their  sex,  age,  starting  GS  level,  current  GS  level,  the  year  in  which 
they  entered  the  Corps,  and  the  year  in  which  they  attained  their  current  GS  level.  From  the  year  in 
which  their  current  GS  level  was  obtained,  a  measure  of  time  in  grade  was  computed;  experience  was 
derived  from  the  year  in  which  they  entered  the  Corps;  and  a  measure  of  promotion  rate  was  obtained 
by  standardizing  current  GS  level  by  groups,  based  on  length  of  experience.  In  essence,  this  procedure 
compared  the  salary  history  of  those  who  entered  the  Corps  at  about  the  same  time.  For  example,  if 
two  individuals  entered  the  Corps  at  the  same  time,  and  one  had  advanced  to  a  GS-14  level  while  the 
other  was  a  GS-I2,  the  former  had  demonstrated  a  higher  level  of  managerial  ability.  Because  each 
experience  category  -  e.g.,  promotion  rate,  time  in  grade  -  was  considered  separately,  the  resulting 
measure,  which  includes  all  the  categories,  would  not  be  expected  to  correlate  with  experience. 

When  the  personnel  at  the  Missouri  River  Division  and  Omaha  District  were  interviewed  to  gain  a 
better  understanding  of  the  operations  of  the  Corps,  as  described  in  Chapter  2,  they  were  also  asked  to 
nominate  a  few  people  they  knew  personally  --  from  either  the  Division  or  District  level  --  who  either 


had  demonstrated  ability  or  demonstrated  potential  to  become  an  outstanding  manager  within  the 
Corps.  These  data  were  then  compiled  and  several  peer  nomination  measures  were  developed.  The 
most  meaningful  measure  from  the  peer  nomination  data  was  the  binary  measure  of  the  person's  hav¬ 
ing  been  nominated  one  or  more  times  versus  the  person's  not  having  been  nominated  at  all.  A  con¬ 
tinuous  measure,  which  was  simply  the  number  of  times  an  individual  was  nominated  (zero  through 
four),  and  a  binary  measure  (nominated  two  or  more  times  versus  nominated  one  or  fewer  times)  did 
not  prove  useful  because  they  did  not  add  anything  to  the  results  obtained  with  the  first  nomination 
measure. 

The  promotion  rate  measure  and  the  peer  nomination  measure  were  both  independent  indications 
of  ability  to  function  at  the  higher  GS  levels.  However,  because  so  few  people  were  actually  nominated 
(15  percent  of  the  sample),  no  known  instrument  could  predict  a  measure  with  such  small  variance. 
Therefore,  peer  nomination  data  could  be  used  best  in  combination  with  the  promotion  rate  in  a  com¬ 
posite,  which  was  a  more  comprehensive  measure  of  pei  formance.  The  work  of  Laurent  indicated  that 
this  procedure  could  be  fruitful;  therefore,  the  composite  measure  was  computed.26 

As  mentioned  earlier  in  in  the  main  text,  the  criterion  problem  -  finding  measures  which  ade¬ 
quately  assess  performance  in  a  given  job  or  field  -  is  central  to  an  investigation  of  this  type.  Develop¬ 
ing  such  criteria  of  performance  thoroughly  for  high-level  managers  and  engineers  in  the  Corps  of 
Engineers  would  require  an  investigation  far  beyond  the  scope  of  this  study.  For  this  preliminary  inves¬ 
tigation,  the  promotion  rate  measure  and  the  composite  of  promotion  rate  and  peer  nomination  pro¬ 
vided  meaningful  performance  measures  which  both  fit  within  the  constraints  of  the  study  and  provided 
a  useful  analysis  of  Form  V. 


The  Total  Sample  Key  —  the  Biographical  Correlates  of  Overall  Managerial  Performance 

To  predict  the  composite  performance  measure  for  the  biographical  items,  the  item  analysis 
(described  above)  using  triple  cross-validation  yielded  three  scoring  keys,  each  based  on  two  thirds  of 
the  total  sample.  While  these  keys  were  very  similar,  they  were  not  identical.  For  discussion  -  and  to 
yield  a  key  which  could  be  used  in  any  future  application  of  Form  V  -  a  scoring  key  for  the  biographi¬ 
cal  items  was  generated  based  on  the  total  sample  of  219  employees.  While  technically  this  key  was  not 
cross-validated,  since  it  was  based  on  the  total  sample  rather  than  two  thirds  as  were  the  three  cross- 
validated  keys,  the  total  sample  key  would  be  more  stable  than  any  of  the  subsample  keys.  Therefore, 
it  is  the  most  logical  key  to  be  applied  to  other,  independent  samples.  This  discussion  describes  the 
items  within  the  scoring  key  and  provides  information  about  how  the  responses  of  high  scoring 
managers  on  the  performance  composite  differed  from  the  responses  of  low  scoring  managers. 

Of  the  subjective  conceptual  categories  previously  described  in  this  appendix,  those  of  Communi¬ 
cations,  Intelligence,  and  items  derived  from  the  Job  Activities  Description  provided  the  highest  per¬ 
centages  of  valid  items.  For  example,  from  the  Communications  area,  persons  with  higher  managerial 
performance  indicated  that  several  times  they  had  delivered  a  speech  before  50  or  more  persons, 
whereas  lower  scoring  managers  indicated  they  never  had.  High-performing  managers  described  their 
public  speaking  ability  as  very  good,  while  low-scoring  managers  tended  to  indicate  that  their  public 
speaking  ability  was  marginal.  More  effective  managers  indicated  that,  whenever  possible,  they  made 
broad  outlines  when  writing  reports,  directives,  and  manuals,  rather  than  avoiding  such  paperwork  or 
taking  a  more  active  role  in  writing  the  actual  text.  High-performing  managers  typically  responded  that 
writing  reports  was  very  easy;  low  performing  managers  indicated  that  it  was  generally  rather  difficult. 
When  asked  which  was  the  stronger  asset,  high  scoring  managers  indicated  communicating  effectively 
with  others  as  opposed  to  following  through  and  completing  requirements. 

Two  items  assigned  to  the  Intelligence  category  asked  for  information  about  how  many  activities 
the  person  would  like  to  manage  at  a  time.  High-performing  managers  typically  indicated  that  they 
would  like  to  have  many  things  going  on  simultaneously.  Individuals  with  low  scores  on  the  perfor¬ 
mance  measure  tended  to  indicate  that  they  would  rather  focus  on  a  few  things  at  a  time.  The  more 
effective  managers  tended  to  describe  their  ability  to  think  on  their  feet  as  very  good  and  also  indicated 
that  they  would  describe  themselves  more  as  quick  witted,  rather  than  as  methodically  thorough.  When 

16  H.  Laurent,  ‘Cross-Cultural,  Cross  Validation  of  Empirically  Validated  Tests,’  Journal  of  Applied  Psychology,  Vol  54  (1970).  pp 
417-423 


asked  to  describe  how  well  they  could  think  under  pressure,  the  more  effective  managers  tended  to 
indicate  that  they  could  think  very  well  under  such  conditions,  while  the  less  effective  managers  tended 
to  indicate  that  they  were  about  average  under  such  conditions. 

As  described  previously,  the  items  concerned  with  job  activities  presented  a  pair  of  activities  and 
asked  the  respondent  to  indicate  in  which  of  the  pair  he/she  was  more  skilled.  For  example,  one  of  the 
questions  asked  whether  the  respondent  was  more  skilled  in  organizing  work  schedules  or  in  evaluating 
program  outcomes.  The  relationships  of  this  item  with  the  performance  measure  indicated  that  higher 
performing  managers  tended  to  respond  that  they  were  more  skilled  in  evaluating  program  outcomes, 
while  lower  performing  managers  indicated  greater  skill  in  organizing  work  schedules.  More  effective 
managers  tended  to  indicate  greater  skill  in  developing  cooperative  relationships  with  outside  groups 
(clients,  special  interest  groups,  etc  );  whereas  lower  performing  managers  tended  to  indicate  more  skill 
in  consulting  on  complex  engineering  problems.  Other  items  usually  associated  with  more  effective 
managers  were  analyzing  developmental  opportunities  for  subordinates  as  opposed  to  preparing  and 
reviewing  cost  estimates;  leading  meetings  and  discussions  rather  than  organizing  reports  and  manuals, 
helping  others  reach  a  consensus  as  opposed  to  writing  progress  reports  or  project  specifications,  and 
developing  long-range  priorities,  as  opposed  to  preparing  and  reviewing  cost  estimates.  In  general,  the 
analysis  of  the  items  from  this  category  indicated  that  the  more  effective  managers  were  more  skilled  in 
activities  requiring  a  broader  perspective,  requiring  skills  more  management  oriented  than  technically 
oriented,  and  requiring  more  interaction  with  people  than  solitary  activities. 

Two  items  from  the  Achievement  Motivation  category  had  significant  relationships  with  the  per¬ 
formance  measure.  The  responses  to  the  first  of  these  indicated  that  the  higher  performing  managers 
would  prefer  to  have  a  good  deal  of  responsibility  in  a  job  while  less  effective  managers  would  like  to 
have  some  responsibility  but  still  have  someone  responsible  over  them.  Less  effective  managers  tended 
to  respond  that  they  had  an  average  background  in  several  subject  matter  fields  when  asked  to  describe 
the  breadth  of  their  subject  matter  knowledge. 

The  more  effective  managers  indicated  a  greater  degree  of  aggressiveness  and  interest  in  imple¬ 
menting  new  methods,  standing  up  for  their  own  ideas,  etc.  When  asked  if  they  would  try  to  change 
restrictive  regulations  or  policies  which  had  a  negative  influence  on  their  work,  the  more  effective 
managers  responded  "definitely  yes."  The  more  effective  managers  indicated  that  they  were  interested  in 
new  methods  and  encouraged  others  to  alert  them  to  such  innovations,  less  effective  managers  took  a 
more  conservative  approach  to  the  implementation  of  new  methods  or  procedures  within  the  group, 
perhaps  indicating  that  their  first  concern  was  getting  the  immediate  job  done. 

As  expected  from  the  interview  discussions,  certain  interpersonal  skills  were  also  important  to 
managerial  effectiveness.  For  example,  when  asked  which  was  most  descriptive  of  them,  the  more 
effective  managers  responded  that  they  enjoy  frequent  contact  and  discussion  with  many  different  peo¬ 
ple,  while  less  effective  managers  liked  working  independently.  Less  effective  managers  indicated  that 
they  would  rarely  let  someone  know  if  they  were  dissatisfied  with  that  person’s  work.  Less  effective 
managers  also  tended  to  respond  that  their  idea  of  an  ideal  job  was  one  which  would  allow  them  to 
work  by  themselves  rather  than  requiring  involvement  with  others.  One  of  the  significant  item- 
criterion  relationships  indicated  that  the  more  effective  managers  would  not  let  getting  along  with  oth¬ 
ers  interfere  with  higher  order  managerial  duties.  The  more  effective  managers  indicated  that  they 
would  more  likely  be  praised  for  coordinating  arrangements  and  activities  rather  than  their  technical 
contributions  or  working  effectively  with  others. 

When  asked  about  their  management  abilities,  the  more  effective  managers  indicated  that  they 
•ere  very  efficient  at  coordinating  a  variety  of  activities  and  duties,  that  they  were  very  good  at  stimu- 
atmg  productive  thought  and  action  in  their  subordinates,  and  that  they  had  the  most  potential  in  forc- 
aftiing  program  development  rather  than  developing  project  specifications.  Managers  who  appeared  less 
Tcitive  on  the  composite  measure  tended  to  indicate  that  they  were  about  average  in  coordinating  a 
«'•<■»>  of  activities  and  duties  and  that  they  had  more  potential  for  developing  project  specifications 
tun  lorecasting  program  development.  The  number  of  business  or  management  courses  com- 

■  %.n*c  graduation  from  college  was  highly  related  to  the  performance  measure. 

■v  -'.ponses  to  the  items  in  the  Management  Judgment  category  indicated  a  greater  willingness 
i  he  more  effective  managers  to  take  an  active  role  in  resolving  potential  conflicts  between 
•  .  *  .r  mo  an  outside  organization,  while  the  less  effective  managers  would  tend  to  notify 


their  supervisor  and  have  him/her  work  out  such  problems.  When  asked  about  their  preference  for 
working  with  clients,  the  more  effective  managers  indicated  that  they  would  assign  someone  to  work 
directly  with  the  client  but  contact  the  client  personally  as  well;  less  effective  managers  indicated  that 
under  similar  circumstances  they  would  work  with  a  client  only  when  it  was  directly  appropriate  to  do 
so.  Those  in  between  tended  to  choose  the  other  responses  which  were  "Make  frequent  personal  con¬ 
tacts  with  clients  to  establish  cooperative  relationships"  and  "Assign  someone  to  work  directly  with  the 
client  to  establish  a  cooperative  relationship."  Those  with  higher  scores  on  the  performance  measure 
tended  to  respond  that,  in  general,  better  managers  spend  most  of  their  time  supervising,  concern 
themselves  mainly  with  indices  of  production,  and  work  with  their  subordinates  to  complete  projects. 
Less  effective  managers,  on  the  other  hand,  concern  themselves  too  much  with  the  technical  responsi¬ 
bilities  of  their  subordinates  rather  than  with  general  supervision.  The  responses  to  the  items  in  the 
Management  Judgment  area  indicated  a  greater  willingness  to  take  on  responsibility  where  appropriate, 
delegate  responsibility  where  appropriate,  and  allocate  time  to  managerial  rather  than  either  technical  or 
"watchdog"  duties. 

Four  of  the  items  from  the  miscellaneous  category  had  significant  relationships  with  the  perfor¬ 
mance  measure.  Since  the  performance  indicator  used  as  a  criterion  for  developing  the  BI  scoring  key 
was  partially  based  on  GS  level,  certain  items  within  the  BI  that  assessed  or  would  be  expected  to  relate 
to  GS  level,  also  related  to  the  criterion.  When  asked  to  describe  the  technical  versus  managerial  duties 
of  their  current  job,  the  more  effective  managers  indicated  that  their  job  was  mostly  managerial  or  was 
divided  between  both  about  equally,  while  less  effective  managers  indicated  that  their  job  was  mostly 
technical.  The  more  effective  managers  indicated  that  they  had  classes  in  the  humanities  or  social  sci¬ 
ences  while  less  effective  managers  indicated  that  they  had  taken  no  such  classes  since  graduation  from 
college.  When  asked  about  their  level  of  involvement  in  the  career  development  of  less  experienced 
personnel,  the  more  effective  managers  indicated  that  they  became  very  involved  in  the  career  develop¬ 
ment  of  their  subordinates  and  considered  that  an  important  part  of  their  job  duties.  Less  effective 
managers  indicated  that  they  were  happy  to  supply  information  when  a  subordinate  came  to  them  with 
questions,  but  that  otherwise  their  involvement  was  somewhat  limited.  When  asked  about  jumping 
from  one  job  activity  to  another,  the  more  effective  managers  indicated  that  requirement  was  often 
stimulating,  while  less  effective  managers  indicated  that  they  would  rather  focus  their  activities 

The  results  obtained  with  the  Breadth  of  Experience  items  produced  some  rather  curious  results. 
For  example,  on  the  items  dealing  with  the  various  divisional  functions  in  the  Corps,  having  more  than 
3  years  of  experience  in  the  operations  function,  having  2  to  3  years  of  experience  in  the  Research  and 
Development  function,  or  having  1  or  2  years  in  the  Engineering  function  were  each  significantly  asso¬ 
ciated  with  higher  performing  managers.  However,  having  had  2  to  3  years  experience  in  the  Planning 
function,  or  having  had  more  than  3  years  experience  in  the  Engineering  function  was  associated  with 
less  effective  managers.  It  is  difficult  to  understand  why  a  good  deal  of  experience  in  Operations  or 
Research  and  Development,  or  a  moderate  amount  of  experience  in  Engineering  is  associated  with  high 
performance,  while  a  lot  of  experience  in  Engineering  or  2  or  3  years  experience  in  Planning  is  associ¬ 
ated  with  poorer  performance.  These  relationships  might  reflect  a  problem  with  the  sample  used,  i.e., 
the  higher  level  managers  in  the  Omaha  District  and  Missouri  River  Division  might  have  been  drawn 
from  Operations  or  Research  and  Development,  while  people  who  stay  in  the  Planning  function  for  a 
long  time  or  in  the  Engineering  function  for  more  than  3  years,  tend  to  stay  at  lower  level  positions. 
The  problem  may  also  be  one  of  numbers;  i.e.,  there  may  be  a  greater  number  of  people  in  the 
Engineering  function,  and  therefore  more  people  would  be  stabilized  in  medium  or  low  level  positions 
for  longer  periods  of  time. 

Of  the  20  items  dealing  with  amount  of  experience  in  the  various  functional  areas,  only  4  had 
significant  relationships  with  the  performance  measure.  The  higher  performing  managers  indicated  that 
they  had  little  or  no  experience  in  the  specifications/estimates  area,  but  had  more  than  3  years  experi¬ 
ence  in  programming/budgeting,  more  than  3  years  experience  in  public  information/education,  or 
more  than  3  years  experience  in  administration/ management.  The  lower  performing  managers  indi¬ 
cated  that  they  had  little  or  no  experience  in  administration/management.  When  asked  in  how  many  of 
the  20  functional  areas  they  had  had  1  or  more  years  of  experience,  the  most  effective  managers  tended 
to  respond  that  they  had  such  experience  in  7  or  8  of  these  areas,  indicating  a  good  deal  of  breadth. 


APPENDIX  E: 


MANUAL  FOR  THE  INTERVIEWING  GUIDE 


Introduction 

This  manual  should  be  helpful  in  the  selection  and  promotion  process  for  management  positions 
at  GS-13  and  above  in  the  Corps  of  Engineers.  Interviewing  is  a  complex  process  requiring  sophistica¬ 
tion  and  experience.  Though  basically  the  process  is  used  to  exchange  information,  it  can  contribute  to 
placing  a  more  highly  qualified  individual  into  a  vacant  position,  which,  in  turn,  can  contribute  to 
organizational  effectiveness,  to  the  satisfaction  of  the  individual,  and  to  the  satisfaction  of  the 
individual’s  subordinate  employees.  Therefore  during  the  interview  the  interviewer  should  not  do  most 
of  the  talking,  but  should  provide  ample  opportunities  for  the  applicant  to  talk  about  his/her  capabilities 
and  previous  experiences  that  are  relevant  to  the  proposed  position. 


Preparation  for  the  Interview 

Before  any  interview,  some  preparation  time  should  be  devoted  to  reviewing  relevant  facts  to  be 
covered  during  the  interview.  The  better  prepared  the  interviewer  is,  the  more  relevant  will  be  the 
information  obtained  from  the  interview.  A  few  important  points  should  be  covered  before  the  inter¬ 
view: 

1 .  The  interviewer  should  review  the  job  requirements  important  for  success  in  the  open  posi¬ 
tion.  Comments  from  supervisors  or  information  obtained  from  job  descriptions  may  be  so  general  that 
the  exact  nature  of  critical  activities  cannot  be  discerned.  So  the  critical  job  requirements  might  not  be 
adequately  covered  in  the  interview  unless  the  interviewer  can  determine  the  specific  job  requirements. 
For  example,  a  vacancy  announcement  may  have  a  phrase  such  as,  "capable  of  communicating 
effectively,"  without  indicating  whether  the  communication  will  be  oral  or  written,  or  both,  whether  the 
communication  will  be  predominantly  of  a  technical  or  managerial  nature,  or  whether  the  communica¬ 
tion  will  be  with  the  general  public,  or  with  Architectural/Engineering  firms.  Conversing  glibly  and 
explaining  to  a  subordinate  the  demands  of  a  particular  plan  are  two  different  communications  skills. 

2.  The  questions  about  specific  job  requirements  should  generally  be  asked  of  every  person  inter¬ 
viewed.  Consistency  is  equivalent  to  fairness  in  this  case.  Therefore,  before  the  first  candidate  is  inter¬ 
viewed,  the  interviewer  must  develop  a  set  of  questions  that  will  be  asked  of  each  applicant  for  a  given 
position.  This  does  not  mean  that  relevant,  unique  characteristics  in  an  applicant's  background  cannot 
be  explored;  however,  the  key  questions  covered  during  the  interview  should  be  consistent. 

3.  Relevant  facts  about  the  candidate  should  be  reviewed  from  the  personnel  folder.  Often,  valu¬ 
able  interview  time  is  spent  on  information  available  in  the  person's  file.  Any  additional  information 
about  previous  experience  or  the  characteristics  of  the  applicant  that  needs  clarification  or  expansion 
should  be  noted  for  exploration  during  the  interview. 

The  process  of  preparing  for  an  interview  need  not  be  long,  but  the  preparation  can  make  a  sub¬ 
stantial  difference  in  both  the  conduct  of  the  interview  and  the  information  gained.  Moreover,  if  the 
interviewer  has  prepared  for  the  interview,  the  applicant  will  feel  that  the  interviewer  is  interested  in 
him/her  as  a  person,  and  will  be  able  to  present  him/herself  better  during  the  meeting. 


Conducting  the  Interview 

The  interview  should  be  comfortable  for  both  the  interviewer  and  the  interviewee.  If  the  inter¬ 
viewer  is  well  prepared,  the  stage  will  be  set  for  a  relaxed  and  free  exchange  of  information. 

The  applicant  should  be  aware  of  what  is  expected  in  the  position,  but  the  interviewer  could  go 
over  this,  ensuring  that  the  applicant  understands  the  general  job  duties,  the  important  job  activities, 
work-unit  objectives,  and  the  level  of  performance  expected.  If  the  applicant  has  no  misconceptions 
about  the  position,  he/she  can  respond  in  a  more  concise  and  relevant  manner  to  the  questions  covered 
in  the  interview. 
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In  general,  a  good  strategy  is  to  ask  open-ended  questions  which  provide  an  opportunity  for  the 
applicant  to  describe  him/herself  and  his/her  experience.  Try  to  give  frequent  positive  remarks  when¬ 
ever  the  applicant  responds  in  an  open,  communicative  manner.  Such  comments  as  "I  understand," 
"yes,"  or  "that's  interesting"  will  help  make  the  applicant  feel  at  ease  and  direct  the  applicant  toward 
more  complete  remarks. 

Obtaining  relatively  complete  answers  about  issues  of  key  importance  is  a  prime  concern  of  the 
interviewer.  In  other  words,  do  not  hesitate  to  probe  further  to  gain  additional  information.  Such 
remarks  as  "can  you  elaborate,"  "in  what  way,"  "how  did  that  come  about,"  or  "tell  me  more  about  this" 
will  help  provide  information  to  supplement  brief,  or  in  other  ways  incomplete,  or  perhaps  evasive 
answers. 

If  the  interviewer  is  thoroughly  familiar  with  the  questions  in  the  Interviewing  Guide,  he/she  can 
use  the  review  of  job  experience  as  a  natural  opening  to  pursue  different  questions  concerning 
managerial  activities.  As  opportunities  permit,  questions  can  be  used  from  any  place  in  the  Interview¬ 
ing  Guide  that  naturally  follow  from  previous  comments  and  fit  smoothly  into  the  conversation.  With 
some  practice,  this  is  fairly  straightforward  and  easy,  as  one  topic  naturally  leads  to  another. 

Again,  each  applicant  for  a  particular  position  should  be  given  the  opportunity  to  answer  the  same 
questions.  Both  the  order  and  the  content  can  vary  to  some  extent,  but  there  should  be  some  coverage 
of  the  Interviewing  Guide’s  areas.  This  will  permit  the  development  of  scores  that  help  in  interpreting 
the  interview  results. 

The  interview  should  be  concluded  by  summarizing  for  the  applicant  the  steps  to  be  taken  in 
reaching  a  selection  decision.  She/he  should  be  told  when  a  decision  will  be  made  and,  when  possible, 
provided  some  information  about  her/his  potential  standing.  If  there  are  candidates  who  are  clearly 
more  qualified,  the  interviewer  should  state  that  there  is  substantial  competition  for  the  position,  and 
that  some  of  the  individuals  appear  to  be  particularly  well  qualified.  In  any  case,  the  candidate  should 
be  promised  that  she/he  will  be  notified  as  soon  as  a  selection  decision  has  been  made.  Finally,  the 
interviewer  should  conclude  by  thanking  the  applicant  for  considering  the  position  and  applying  for  it. 


The  Interviewing  Guide 

The  Interviewing  Guide  has  three  sections:  the  first  concerned  with  the  applicant’s  managerial 
competence,  the  second  with  job-specific  questions,  and  the  third  with  the  interviewer’s  subjective, 
overall  impressions  of  the  interviewee. 

1.  Managerial  Components.  The  questions  in  this  section  are  designed  to  explore  the  applicant's 
potential,  as  well  as  proven  managerial  capabilities.  Assessing  managerial  skills  may  be  difficult  since 
the  applicant’s  previous  positions  may  not  have  provided  enough  opportunity  to  develop  these  skills 
Each  item  in  the  managerial  section  contains  a  general,  open-ended  question,  and  short  phrases  which 
can  be  used  to  record,  in  part,  the  interviewee’s  responses.  The  phrases  are  intended  to  give  the  inter¬ 
viewer  some  indication  of  the  types  of  important  responses  that  should  be  sought  in  the  interview  As 
the  interview  proceeds,  the  interviewer  is  to  evaluate  the  applicant  on  each  of  the  more  detailed  points 
listed  under  the  questions.  The  interviewer  should  develop  his/her  own  unobtrusive  method  for  doing 
this,  but  that  system  should  indicate  whether  the  applicant  showed  a  strength  or  weakness  on  the  point, 
was  average,  or  could  not  be  evaluated.  For  example,  the  interviewer  might  use  +  ,  -,  0,  or  blank, 
respectively,  to  record  the  evaluations  on  each  point.  After  the  interview  is  concluded,  the  interviewer 
should  make  a  rating  for  each  of  the  questions  covered  in  the  Managerial  Components  section  The 
rating  on  each  of  the  major  questions  should  carefully  consider  the  evaluations  made  during  the  inter¬ 
view  on  the  more  detailed  points 

2.  Job-Specific  Questions.  The  specific  technical  requirements  of  the  job  should  be  written  in  the 
space  provided  Specific  requirements  are  not  preprinted  on  the  form  because  of  the  complexity  of  the 
engineering  positions  in  the  Corps.  Space  is  provided  for  rating  the  applicant’s  remarks  about  his/her 
competence  in  key  job  requirements  and  summarizing  the  applicant’s  experience  and  judged  capability 
for  each  of  the  key  job  requirements.  The  rating  can  be  checked  off  during  the  interview,  but  there 
should  be  enough  documentation  of  the  remarks  so  that  the  material  can  be  reviewed  later 

3.  Interviewer’s  Impressions.  This  portion  of  the  Interviewing  Guide  contains  items  that  the 
interviewer  is  to  answer  after  the  interview  has  concluded.  These  are  relatively  subjective  ratings  on 


84 


the  part  of  the  interviewer,  and  are  based  on  his/her  impressions  during  the  interview.  Nonetheless, 
these  impressions  can  be  important  indicators  of  the  potential  of  the  individual  and  should  receive 
attention  during  the  selection  process. 


Scoring  of  the  Interview 

The  items  in  the  Technical  Job  Requirements  section,  the  major  questions  covered  in  the 
Managerial  Components  section,  and  the  items  of  the  Interviewer’s  Impressions  section  are  all  to  be 
scored  with  the  following  scale: 

Blank  =  Insufficient  evidence,  applicant  cannot  be  rated 

1  =  Marginal  competence 

2  =  Some  potential,  further  development  needed 

3  =  Satisfactory  competence 

4  =  Very  satisfactory  competence 

5  =  Outstanding  capability. 

The  numerical  rating  for  each  item  should  be  marked  in  the  space  provided.  Throughout  this  rat¬ 
ing  process,  the  interviewer  must  take  care  that  each  score  consider  only  the  item  at  hand  and  not  an 
overall  feeling  about  the  interviewee.  No  applicant  should  be  rated  at  the  same  level  on  all  items. 
Therefore,  the  interviewer  should  observe  carefully  the  strengths  and  weaknesses  of  all  applicants. 

The  total  of  the  ratings  for  each  section  is  entered  at  the  end  of  the  section,  and.  to  obtain  the 
average,  is  divided  by  the  number  of  items  on  which  the  sum  is  based.  This  division  is  done  so  that 
the  applicants  can  be  compared  more  readily.  An  applicant  should  not  be  penalized  if  insufficient  evi¬ 
dence  is  available  for  the  interviewer  to  make  a  rating;  therefore,  blanks  are  not  counted  in  the  compu¬ 
tation  of  the  average. 

An  overall  score  for  the  applicant  can  be  obtained  by  totaling  the  average  scores  obtained  for  each 
of  the  three  sections.  Space  for  this  is  provided  on  the  last  page  of  the  form. 


Summarizing  the  Results 

The  interviewer  should  summarize  the  key  remarks  that  will  help  document  what  happened  dur¬ 
ing  the  interview.  This  should  probably  be  done  shortly  after  the  interview  has  concluded.  Any  ques¬ 
tions  that  need  further  follow-up  (with  the  applicant's  present  supervisor,  coworkers,  etc  )  should  also 
be  noted. 


Using  the  Results 

After  all  the  interviews  with  potential  employees  have  been  held,  applicants  can  be  compared  on 
the  basis  of  the  scoring  system  devised  for  the  interviews.  However,  selecting  the  person  to  fill  the 
position  solely  on  the  basis  of  the  interview  would  be  ineffective  and  a  misuse  of  the  interview  The 
information  gained  from  this  process  should  be  used  in  combination  with  all  the  other  sources  of  infor¬ 
mation  to  make  the  selection.  The  interviewer  should  be  aware  that  some  individuals  do  not  come 
across  well  in  an  interview,  or  that  an  individual  might  respond  much  better  at  another  time  The 
interviewer  should  also  be  aware  that  she/he  functions  better  on  different  occasions,  and  this.  too. 
might  have  an  effect  on  the  results. 

The  use  of  a  structured  interview  can  be  an  asset  in  the  selection  process  because  of  the  kinds  of 
information  that  might  not  be  available  in  any  other  way  The  individual  is  the  best  source  for  certain 
information  about  her/himself,  and  the  interview,  if  structured  correctly,  can  help  obtain  this  informa¬ 
tion. 
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APPENDIX  F 


FEEDBACK  OF  MAS  RESULTS 


The  system  offers  two  kinds  of  scoring  information,  a  percentile  rank  score  and  the  criterion- 
referenced  score  (percent  favorable) ,  so  that  both  comparative  and  absolute  information  are  available  to 
facilitate  effective  interpretation  of  the  results.  After  the  screening  process  is  completed,  the  MAS  sys¬ 
tem  provides  each  supervisor  of  five  or  more  employees  a  computer-generated  feedback  report  describ¬ 
ing  -  in  terms  of  achievements  and  opportunities  for  improvement  -  the  state  of  human  resource 
management  for  his/her  work  group.  Higher  levels  of  management  receive  two  feedback  reports  -  one 
from  subordinate  supervisors  (if  there  are  five  or  more)  and  a  combined  report  for  all  employees  subor¬ 
dinate  to  him/her. 

Percent  Favorable  Scores 

Results  in  Table  FI  show  the  percent  favorable  scores  in  the  two  columns  on  the  right  side  of  the 
page.  These  scores  show  the  actual  percentage  of  favorable  responses  chosen  by  employees  to  describe 
characteristics  of  management  in  their  organizational  unit.  The  column  titled  COMPONENT  is  the 
group  of  scores  for  the  total  organization;  the  column  titled  ORGANIZATION  is  the  group  of  scores 
for  the  data  base  (in  this  case  all  three  Corps  sites)  which  the  component  is  being  compared  against  In 
the  event  that  an  organizational  unit  has  25  or  more  employees  -  e  g.,  a  division  like  Engineering  or 
Operations  --  the  unit  will  also  get  a  column  of  scores  titled  UNIT. 

The  percent  favorable  scores  are  typically  built  on  the  two  most  favorable  responses  for  each 
question.  The  number  of  positive  responses  chosen  across  (he  questions  in  a  score  area  is  shown  as  a 
percentage  of  the  number  of  positive  responses  as  possible  for  that  area.  For  example,  in  the  Table  FI 

Table  F! 

MAS  Feedback 
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the  total  Corps  of  Engineers  sample  received  a  65  percent  favorable  score  for  Fairness  of  Management. 
This  indicates  that  the  Corps  employees  chose  65  percent  of  the  favorable  responses  to  describe  the 
Fairness  of  Management  across  all  three  Corps  sites.  At  the  organization  where  this  feedback  form  was 
sent,  the  percent  favorable  score  was  68,  indicating  that  the  employees  at  this  one  Corps  site  were  more 
positive  than  the  total  Corps  sample  about  the  Fairness  of  Management  in  their  own  organization 

An  examination  of  the  scores  for  a  single  organization  reveals  a  variable  pattern  of  scores  across 
the  management  areas.  While  some  organizations  may  tend  to  have  somewhat  high  or  low  scores,  no 
organization  has  only  high  or  only  low  scores.  For  example,  in  the  organization  represented  in  Table 
FI,  the  highest  score  was  a  73  on  Operational  Efficiency  and  a  low  score  of  20  on  Opportunity  for  Pro¬ 
motions,  with  many  scores  in  the  40s,  50s  and  60s.  This  same  variation  across  score  areas  can  be  seen 
for  any  organization. 

Percentile  Rank  Scores 

This  scoring  method  takes  into  account  positive,  neutral,  and  negative  responses  and  permits  the 
supervisor  to  see  how  well  a  particular  unit  did  in  comparison  with  all  the  units  in  the  organization.  In 
this  scoring  method  the  total  organization  is  always  the  50th  percentile.  If  a  unit  has  a  score  of  50  or 
better  in  any  management  area,  that  unit  scored  as  well  or  better  than  the  average  of  all  the  employees 
in  the  organization.  These  scores  are  presented  in  the  two  left-hand  columns  on  the  feedback  report. 
In  the  column  titled  ORGANIZATION  are  the  scores  for  the  entire,  single  organization  but  not  for  all 
three  Corps  sites.  The  next  column  titled  COMPONENT  presents  the  scores  for  the  organizational  unit 
in  which  the  six  employees  work  (such  as  a  section,  team,  branch). 

The  supervisor  looks  at  the  scores  for  the  organization  and  compares  his/her  component  scores  to 
them.  For  example,  in  Delegation  of  Authority,  the  percentile  rank  of  the  entire  organization  was  24, 
whereas  the  score  for  the  supervisor  of  this  particular  unit  was  35,  or  1 1  percentile  points  higher  than 
the  organization. 

These  are  comparative  scores ,  primarily  used  by  individual  managers  to  see  how  they  compare 
with  the  entire  organization.  Any  scores  within  the  range  of  40  to  60  generally  should  not  be  con¬ 
sidered  either  particularly  high  or  low  since  they  are  close  to  the  organization's  average  Scores  in  the 
30s  are  potential  red  flags  and  should  be  noted  as  areas  needing  improvement  Scores  lower  than  30  - 
particularly  those  in  the  teens  -  indicate  severe  problems. 

Interpretation  of  the  MAS  Scores 

Since  the  two  scoring  systems  provide  different  information,  each  can  be  looked  at  individually 
but  provide  more  complete  information  when  examined  together.  The  percentile  rank  score  provides 
comparative  information  on  how  a  unit  is  viewed  by  employees  relative  to  all  the  units  in  the  orgamza- 
,ion.  The  percent  favorable  scores  show  the  actual  percentage  of  favorable  responses  made  by  employ 
ees  in  the  organization. 

In  analyzing  the  two  scores,  the  supervisor  should  look  first  at  the  percent  of  favorable  responses 
If  the  percentage  for  the  unit  is  /owin  a  given  management  area  (for  example  30  percent),  the  supervi 
sor  should  devote  some  attention  to  that  area.  Even  if  the  unit  were  somewhat  above  average  on  per¬ 
centile  rank,  improvement  would  still  be  needed  because  of  the  low  percentage  of  favorable  responses 

On  the  other  hand,  if  the  percent  favorable  score  for  the  unit  is  high  (70  percent  or  higher),  then 
a  relatively  low  percentile  rank  score  for  the  unit  may  not  be  a  cause  for  concern  The  overall  level  of 
achievement  might  be  so  high  in  the  organization  that  a  low  percentile  rank  might  be  caused  by  the 
high  overall  standing  of  the  organization  rather  than  dissatisfaction  within  a  particular  work  group 

The  supervisor  must  review  the  results  obtained  and  decide  which,  if  any.  of  the  management 
areas  need  attention.  Whether  any  attention  should  be  devoted  to  a  particular  area  would  depend  on 
the  score  obtained  and  on  the  importance  of  the  score  area  to  overall  effective  performance  in  the  unit 
Only  a  few  areas  should  be  selected  for  improvement  since  changes  in  a  large  number  would  be  difficult 
to  achieve  within  a  reasonable  period  of  time.  Generally  speaking,  areas  of  greatest  concern  will  be 
those  in  which  the  work  group  has  less  than  50  percent  favorable  responses  and  for  which  the  unit  falls 
below  the  organization's  average. 
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APPENDIX  G: 


USING  THE  MANAGEMENT  SURVEY  ACTION  PLAN 


The  MAS  has  been  administered  to  over  30,000  employees  in  various  governmental  and  industrial 
settings  and  has  demonstrated  significant  relationships  with  productivity  of  work  units  in  certain  govern¬ 
mental  operations.  That  is,  as  the  MAS  scores  of  a  unit  increase,  the  productivity  of  the  unit  also 
increases.  The  MAS  can  generally  be  expected  to  provide  valid  and  reliable  information  and  to  help 
identify  problem  areas.  A  crucial  step  in  increasing  productivity  and  employee  satisfaction  is  the 
development  of  an  action  plan  to  correct  those  areas  in  which  there  are  opportunities  for  improvement 
Since  information  from  the  MAS  is  helpful  only  if  the  supervisor  acts  upon  it,  the  purpose  of  the  action 
plan  is  to  help  promote  effective  use  of  the  information. 

The  sample  Action  Plan  completed  in  Table  1 1  shows  how  a  supervisor  used  the  MAS  computer 
feedback  results  in  an  attempt  to  improve  the  management  practices  of  his/her  work  group  An  Action 
Plan  would  differ  for  each  individual  work  group,  depending  on  the  particular  needs  and  objecm  ;s  of 
the  unit.  However,  the  supervisor  should  consider,  as  a  minimum,  each  of  the  steps  listed  on  the  sam¬ 
ple  Action  Plan. 

The  process  outlined  in  the  various  steps  should  be  repeated  for  each  problem  area  identified  for 
the  work  group.  Typically,  not  more  than  three  problem  area  ,  should  be  considered  for  action  plans 
Attempting  to  improve  more  areas  would  tend  to  spread  the  effort  too  thin,  require  more  time  than  is 
available,  and  decrease  the  probability  that  the  effort  would  be  successful. 

1.  Definition  of  Problem  and  Related  MAS  Scores.  The  supervisor  should  look  very  closely  at 
the  scores  on  his/her  feedback  report  to  determine  any  areas  of  concern.  These  may  be  the  lowest 
scores,  or  may  be  scores  that  are  not  as  high  as  the  supervisor  thinks  they  should  be  Some  low  scores 
may  not  be  within  the  supervisor’s  power  to  change,  or  may  not  be  in  an  area  important  to  the  function 
of  the  work  group.  In  making  this  determination,  the  supervisor  should  consider  everything  that 
he/she  knows  about  the  work  group  -  the  personnel,  the  objectives,  the  situation.  Employees  can  usu¬ 
ally  offer  a  variety  of  helpful  suggestions  for  defining  and  solving  problems  within  their  work  group 
The  MAS  will  help  identify  problem  areas,  but  to  help  further  define  the  problems,  the  supervisor  may 
want  to  obtain  from  the  Handbook  for  Supervisors  a  list  of  the  other  MAS  scores  that  are  highly  related 
to  the  score  of  concern. 

2  Task  Group  Formed  The  next  step  is  to  decide  whether  a  task  group  is  needed  to  recommend 
new  activities,  alter  procedures,  delegate  authority,  or  change  work  assignments  to  improve  the  problem 
score  area.  The  task  group  would  be  composed  of  knowledgeable  individuals  within  the  organizational 
unit,  and,  if  appropriate,  individuals  from  other  groups  which  work  with  the  unit  in  question.  Resource 
people  who  can  provide  specific  types  of  assistance  can  also  be  consulted  The  ideal  would  be  to 
develop  a  systematic  approach  to  problem  solving  by  calling  on  a  knowledgeable  and  dependable  group 
of  consultants  The  task  group  will  typically  include  some  individuals  who  have  the  authority  to  imple 
ment  the  Action  Plan,  and  some  employees  who  will  be  affected  by  the  plan  However,  the  group 
should  be  no  larger  than  necessary,  and  should  establish  early  reporting  dates 

3  Problem-Solving  Activities  The  supervisor  and/or  the  task  group,  if  formed,  should  next  for¬ 
mulate  new  activities  and  possible  solutions  that  are  appropriate  to  the  problem  and  to  the  objectives  of 
the  work  group  A  resource  for  this  part  of  the  process  is  the  section  of  the  Handbook  for  Supervisors 
devoted  to  the  problem  area  or  score  of  concern  This  section  should  be  used  to  find  possible  action 
strategies  that  could  improve  management  performance  The  sections  for  the  related  MAS  score  areas 
should  also  be  consulted  The  action  strategies  listed  should  give  the  supervisor  and/or  the  task  group 
ideas  about  how  to  attack  the  problem  The  alternate  strategies  should  be  discussed,  and  some  priority 
should  be  assigned  to  each  The  supervisor  has  to  decide  for  him/herself  which  of  the  activities  are  to 
be  implemented 

4  Planned  Follow-Up  The  MAS  Action  Plan  should  have  a  built-in  program  for  supplying  to 
the  supervisor  and/or  the  task  group  information  about  the  impact  of  the  problem-solving  activities  on 
the  problem  areas  identified  This  information  can  be  sought  in  meetings  with  the  entire  work  group, 
in  individual  sessions  with  members  of  the  work  group,  or  in  periodic  written  reports  Such  feedback 
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will  tell  the  supervisor  and/or  the  task  group  how  effective  the  problem-solving  activities  have  been  and 
whether  further  changes  need  to  be  made. 

5.  Responsibility.  This  portion  of  the  MAS  Action  Plan  lists  the  person(s)  responsible  for  seeing 
that  the  problem-solving  activities  are  carried  out,  for  evaluating  the  results  of  implementing  the  activi¬ 
ties,  and  for  making  any  further  changes  in  the  action  strategy.  Typically,  unless  someone  has  this 
responsibility,  the  change  will  not  be  effected. 

6.  Resource  People.  The  supervisor  is  encouraged,  whenever  difficulties  are  encountered,  to  use 
resource  people  within  the  organization  -  people  from  Training,  Management  Analysis,  Personnel,  etc. 
These  individuals  can  use  their  special  training  and  experience  to  help  resolve  problems,  and  can  also 
provide  information,  consultation,  support,  and  other  services  to  promote  more  effective  management 
practices. 
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